Automatic Youtube video and stream archiver.

Set configuration by setting up environment variables.

Example of fairly common configuration is:

export YTFEED_VERIFICATION_TOKEN='sometoken'
export YTFEED_VERIFICATION_SECRET='somesecret' 
export YTFEED_RESUB_TOPIC="https://www.youtube.com/xml/feeds/videos.xml?channel_id=channelid"
export YTFEED_RESUB_CALLBACK_ADDR='https://my.server.addr/path'
export YTFEED_STORAGE_BACKEND='s3'
export YTFEED_S3_ENDPOINT='my.s3.endpoint'
export YTFEED_S3_ACCESS_KEY_ID='acceskeyid'
export YTFEED_S3_SECRET_ACCESS_KEY='secretaccesskey'
export YTFEED_S3_BUCKET_NAME='ytfeed'
export YTFEED_YOUTUBE_API_KEY='myyoutubeapikey'
export YTFEED_RESUB_INTERVAL='24h'

export YTFEED_VIDEO_FORMAT_QUALITY='720'
export YTFEED_VIDEO_FORMAT_EXTENSION='webm'
export YTFEED_VIDEO_DOWNLOAD_RETRY_DELAY='5m'

export YTFEED_BOLTDB_PATH='/home/ytfeed/schedule.db'
export YTFEED_STREAM_SCHEDULER_WORKER_INTERVAL='1m'

Your ol' plain go build cmd/ytfeed/main.go

• To run s3 test or complete suite test, you need to run test_s3.sh for it to start a Minio server. You can stop it or delete it after testing by running test_s3_teardown.sh
• You also need to set YTFEED_YOUTUBE_API_KEY, YTFEED_YOUTUBE_VIDEO_ID, and YTFEED_YOUTUBE_VIDEO_URL to do a complete suite test.
• You can redo failed download by sending a POST request with the XML message in the log as the body.
• If you are planning to download live broadcast, the downloaded video will be in the format mp4 regardless of your extension config.
• If you send SIGINT while ytfeed is downloading video, it will only kills the immediate youtube-dl process, you have to make sure ffmpeg process youtube-dl spawned is also killed and the resulting temporary directory and file removed.
• If the program got killed because of OOM, you can turn on swap file if you cannot raise the machine's memory. Or you can PR me a better way to handle the upload.
• If you want to use stream scheduler, make sure file that is pointed at YTFEED_BOLTDB_PATH is already exists. You can create the file using touch $YTFEED_BOLTDB_PATH command.

• You MUST install youtube-dl first and MUST be available at $PATH.
• You MUST install ffmpeg too for this to work.
• You MUST* run the program first and make sure it could be reached from the internet, then go here to subscribe to a channel.
• From the page in the URL above:
  • Fill Callback URL field with the address of your program, for example http://my.ip.address:8123. This must be the same as the value of YTFEED_RESUB_CALLBACK_ADDR.
  • Fill Topic URL field to be the same as your YTFEED_RESUB_TOPIC.
  • Choose Verify type field as synchronous.
  • Choose Mode field as subscribe.
  • Fill Verify token field to be the same as your YTFEED_VERIFICATION_TOKEN.
  • Fill HMAC secret field to be the same as your YTFEED_VERIFICATION_SECRET.
  • Let Lease seconds field to be empty.
  • Press Do it button.
  • There should be a log that says something like INFO[0635] Subscribed to topic https://www.youtube.com/xml/feeds/videos.xml?channel_id=channelid with challenge 4776706698370686416.

Set the required environment variables and run the binary like ./ytfeed and you'll see log messages if it runs.

• Keep your code super simple and clean.
• Add comments to your code if you have to explain what your code is doing.
• Make sure you have Commitizen CLI installed and your commit message must be written through git cz.
• Pull request to branch other than development will be rejected.