R-yan/l1ving_youtube-dl
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge branch 'master' of https://github.com/rg3/youtube-dl into multipart_videos

Browse Source
This commit is contained in:
Mark Lee 2015-03-15 21:18:15 -07:00
commit 3d89a7b54b
71 changed files with 1931 additions and 599 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.travis.yml
View File

@ -2,6 +2,7 @@ language: python
python: python:
- "2.6" - "2.6"
- "2.7" - "2.7"
- "3.2"
- "3.3" - "3.3"
- "3.4" - "3.4"
before_install: before_install:

3
AUTHORS
View File

@ -113,3 +113,6 @@ Robin de Rooij
Ryan Schmidt Ryan Schmidt
Leslie P. Polzer Leslie P. Polzer
Duncan Keall Duncan Keall
Alexander Mamay
Devin J. Pohly
Eduardo Ferro Aldama

4
CONTRIBUTING.md
View File

@ -18,7 +18,9 @@ If your report is shorter than two lines, it is almost certainly missing some of
For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the -v flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information. For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the -v flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information.
Site support requests **must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL. If your server has multiple IPs or you suspect censorship, adding --call-home may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL.
### Are you using the latest version? ### Are you using the latest version?

434
README.md
View File

@ -47,211 +47,109 @@ which means you can modify it, redistribute it or use it however you like.
# OPTIONS # OPTIONS
-h, --help print this help text and exit -h, --help print this help text and exit
--version print program version and exit --version print program version and exit
-U, --update update this program to latest version. Make -U, --update update this program to latest version. Make sure that you have sufficient permissions (run with sudo if needed)
sure that you have sufficient permissions -i, --ignore-errors continue on download errors, for example to skip unavailable videos in a playlist
(run with sudo if needed) --abort-on-error Abort downloading of further videos (in the playlist or the command line) if an error occurs
-i, --ignore-errors continue on download errors, for example to
skip unavailable videos in a playlist
--abort-on-error Abort downloading of further videos (in the
playlist or the command line) if an error
occurs
--dump-user-agent display the current browser identification --dump-user-agent display the current browser identification
--list-extractors List all supported extractors and the URLs --list-extractors List all supported extractors and the URLs they would handle
they would handle --extractor-descriptions Output descriptions of all supported extractors
--extractor-descriptions Output descriptions of all supported --default-search PREFIX Use this prefix for unqualified URLs. For example "gvsearch2:" downloads two videos from google videos for youtube-dl "large apple".
extractors Use the value "auto" to let youtube-dl guess ("auto_warning" to emit a warning when guessing). "error" just throws an error. The
--default-search PREFIX Use this prefix for unqualified URLs. For default value "fixup_error" repairs broken URLs, but emits an error if this is not possible instead of searching.
example "gvsearch2:" downloads two videos --ignore-config Do not read configuration files. When given in the global configuration file /etc/youtube-dl.conf: Do not read the user configuration
from google videos for youtube-dl "large in ~/.config/youtube-dl/config (%APPDATA%/youtube-dl/config.txt on Windows)
apple". Use the value "auto" to let --flat-playlist Do not extract the videos of a playlist, only list them.
youtube-dl guess ("auto_warning" to emit a
warning when guessing). "error" just throws
an error. The default value "fixup_error"
repairs broken URLs, but emits an error if
this is not possible instead of searching.
--ignore-config Do not read configuration files. When given
in the global configuration file /etc
/youtube-dl.conf: Do not read the user
configuration in ~/.config/youtube-
dl/config (%APPDATA%/youtube-dl/config.txt
on Windows)
--flat-playlist Do not extract the videos of a playlist,
only list them.
--no-color Do not emit color codes in output. --no-color Do not emit color codes in output.
## Network Options: ## Network Options:
--proxy URL Use the specified HTTP/HTTPS proxy. Pass in --proxy URL Use the specified HTTP/HTTPS proxy. Pass in an empty string (--proxy "") for direct connection
an empty string (--proxy "") for direct
connection
--socket-timeout SECONDS Time to wait before giving up, in seconds --socket-timeout SECONDS Time to wait before giving up, in seconds
--source-address IP Client-side IP address to bind to --source-address IP Client-side IP address to bind to (experimental)
(experimental) -4, --force-ipv4 Make all connections via IPv4 (experimental)
-4, --force-ipv4 Make all connections via IPv4 -6, --force-ipv6 Make all connections via IPv6 (experimental)
(experimental) --cn-verification-proxy URL Use this proxy to verify the IP address for some Chinese sites. The default proxy specified by --proxy (or none, if the options is
-6, --force-ipv6 Make all connections via IPv6 not present) is used for the actual downloading. (experimental)
(experimental)
## Video Selection: ## Video Selection:
--playlist-start NUMBER playlist video to start at (default is 1) --playlist-start NUMBER playlist video to start at (default is 1)
--playlist-end NUMBER playlist video to end at (default is last) --playlist-end NUMBER playlist video to end at (default is last)
--playlist-items ITEM_SPEC playlist video items to download. Specify --playlist-items ITEM_SPEC playlist video items to download. Specify indices of the videos in the playlist seperated by commas like: "--playlist-items 1,2,5,8"
indices of the videos in the playlist if you want to download videos indexed 1, 2, 5, 8 in the playlist. You can specify range: "--playlist-items 1-3,7,10-13", it will
seperated by commas like: "--playlist-items download the videos at index 1, 2, 3, 7, 10, 11, 12 and 13.
1,2,5,8" if you want to download videos --match-title REGEX download only matching titles (regex or caseless sub-string)
indexed 1, 2, 5, 8 in the playlist. You can --reject-title REGEX skip download for matching titles (regex or caseless sub-string)
specify range: "--playlist-items
1-3,7,10-13", it will download the videos
at index 1, 2, 3, 7, 10, 11, 12 and 13.
--match-title REGEX download only matching titles (regex or
caseless sub-string)
--reject-title REGEX skip download for matching titles (regex or
caseless sub-string)
--max-downloads NUMBER Abort after downloading NUMBER files --max-downloads NUMBER Abort after downloading NUMBER files
--min-filesize SIZE Do not download any videos smaller than --min-filesize SIZE Do not download any videos smaller than SIZE (e.g. 50k or 44.6m)
SIZE (e.g. 50k or 44.6m) --max-filesize SIZE Do not download any videos larger than SIZE (e.g. 50k or 44.6m)
--max-filesize SIZE Do not download any videos larger than SIZE
(e.g. 50k or 44.6m)
--date DATE download only videos uploaded in this date --date DATE download only videos uploaded in this date
--datebefore DATE download only videos uploaded on or before --datebefore DATE download only videos uploaded on or before this date (i.e. inclusive)
this date (i.e. inclusive) --dateafter DATE download only videos uploaded on or after this date (i.e. inclusive)
--dateafter DATE download only videos uploaded on or after --min-views COUNT Do not download any videos with less than COUNT views
this date (i.e. inclusive) --max-views COUNT Do not download any videos with more than COUNT views
--min-views COUNT Do not download any videos with less than --match-filter FILTER (Experimental) Generic video filter. Specify any key (see help for -o for a list of available keys) to match if the key is present,
COUNT views !key to check if the key is not present,key > NUMBER (like "comment_count > 12", also works with >=, <, <=, !=, =) to compare against
--max-views COUNT Do not download any videos with more than a number, and & to require multiple matches. Values which are not known are excluded unless you put a question mark (?) after the
COUNT views operator.For example, to only match videos that have been liked more than 100 times and disliked less than 50 times (or the dislike
--match-filter FILTER (Experimental) Generic video filter. functionality is not available at the given service), but who also have a description, use --match-filter "like_count > 100 &
Specify any key (see help for -o for a list
of available keys) to match if the key is
present, !key to check if the key is not
present,key > NUMBER (like "comment_count >
12", also works with >=, <, <=, !=, =) to
compare against a number, and & to require
multiple matches. Values which are not
known are excluded unless you put a
question mark (?) after the operator.For
example, to only match videos that have
been liked more than 100 times and disliked
less than 50 times (or the dislike
functionality is not available at the given
service), but who also have a description,
use --match-filter "like_count > 100 &
dislike_count <? 50 & description" . dislike_count <? 50 & description" .
--no-playlist If the URL refers to a video and a --no-playlist If the URL refers to a video and a playlist, download only the video.
playlist, download only the video. --yes-playlist If the URL refers to a video and a playlist, download the playlist.
--yes-playlist If the URL refers to a video and a --age-limit YEARS download only videos suitable for the given age
playlist, download the playlist. --download-archive FILE Download only videos not listed in the archive file. Record the IDs of all downloaded videos in it.
--age-limit YEARS download only videos suitable for the given --include-ads Download advertisements as well (experimental)
age
--download-archive FILE Download only videos not listed in the
archive file. Record the IDs of all
downloaded videos in it.
--include-ads Download advertisements as well
(experimental)
## Download Options: ## Download Options:
-r, --rate-limit LIMIT maximum download rate in bytes per second -r, --rate-limit LIMIT maximum download rate in bytes per second (e.g. 50K or 4.2M)
(e.g. 50K or 4.2M) -R, --retries RETRIES number of retries (default is 10), or "infinite".
-R, --retries RETRIES number of retries (default is 10), or --buffer-size SIZE size of download buffer (e.g. 1024 or 16K) (default is 1024)
"infinite". --no-resize-buffer do not automatically adjust the buffer size. By default, the buffer size is automatically resized from an initial value of SIZE.
--buffer-size SIZE size of download buffer (e.g. 1024 or 16K)
(default is 1024)
--no-resize-buffer do not automatically adjust the buffer
size. By default, the buffer size is
automatically resized from an initial value
of SIZE.
--playlist-reverse Download playlist videos in reverse order --playlist-reverse Download playlist videos in reverse order
--xattr-set-filesize (experimental) set file xattribute --xattr-set-filesize (experimental) set file xattribute ytdl.filesize with expected filesize
ytdl.filesize with expected filesize --hls-prefer-native (experimental) Use the native HLS downloader instead of ffmpeg.
--hls-prefer-native (experimental) Use the native HLS --external-downloader COMMAND Use the specified external downloader. Currently supports aria2c,curl,wget
downloader instead of ffmpeg. --external-downloader-args ARGS Give these arguments to the external downloader.
--external-downloader COMMAND (experimental) Use the specified external
downloader. Currently supports
aria2c,curl,wget
## Filesystem Options: ## Filesystem Options:
-a, --batch-file FILE file containing URLs to download ('-' for -a, --batch-file FILE file containing URLs to download ('-' for stdin)
stdin)
--id use only video ID in file name --id use only video ID in file name
-o, --output TEMPLATE output filename template. Use %(title)s to -o, --output TEMPLATE output filename template. Use %(title)s to get the title, %(uploader)s for the uploader name, %(uploader_id)s for the uploader
get the title, %(uploader)s for the nickname if different, %(autonumber)s to get an automatically incremented number, %(ext)s for the filename extension, %(format)s for
uploader name, %(uploader_id)s for the the format description (like "22 - 1280x720" or "HD"), %(format_id)s for the unique id of the format (like Youtube's itags: "137"),
uploader nickname if different, %(upload_date)s for the upload date (YYYYMMDD), %(extractor)s for the provider (youtube, metacafe, etc), %(id)s for the video id,
%(autonumber)s to get an automatically %(playlist_title)s, %(playlist_id)s, or %(playlist)s (=title if present, ID otherwise) for the playlist the video is in,
incremented number, %(ext)s for the %(playlist_index)s for the position in the playlist. %(height)s and %(width)s for the width and height of the video format.
filename extension, %(format)s for the %(resolution)s for a textual description of the resolution of the video format. %% for a literal percent. Use - to output to stdout.
format description (like "22 - 1280x720" or Can also be used to download to a different directory, for example with -o '/my/downloads/%(uploader)s/%(title)s-%(id)s.%(ext)s' .
"HD"), %(format_id)s for the unique id of --autonumber-size NUMBER Specifies the number of digits in %(autonumber)s when it is present in output filename template or --auto-number option is given
the format (like Youtube's itags: "137"), --restrict-filenames Restrict filenames to only ASCII characters, and avoid "&" and spaces in filenames
%(upload_date)s for the upload date -A, --auto-number [deprecated; use -o "%(autonumber)s-%(title)s.%(ext)s" ] number downloaded files starting from 00000
(YYYYMMDD), %(extractor)s for the provider -t, --title [deprecated] use title in file name (default)
(youtube, metacafe, etc), %(id)s for the
video id, %(playlist_title)s,
%(playlist_id)s, or %(playlist)s (=title if
present, ID otherwise) for the playlist the
video is in, %(playlist_index)s for the
position in the playlist. %(height)s and
%(width)s for the width and height of the
video format. %(resolution)s for a textual
description of the resolution of the video
format. %% for a literal percent. Use - to
output to stdout. Can also be used to
download to a different directory, for
example with -o '/my/downloads/%(uploader)s
/%(title)s-%(id)s.%(ext)s' .
--autonumber-size NUMBER Specifies the number of digits in
%(autonumber)s when it is present in output
filename template or --auto-number option
is given
--restrict-filenames Restrict filenames to only ASCII
characters, and avoid "&" and spaces in
filenames
-A, --auto-number [deprecated; use -o
"%(autonumber)s-%(title)s.%(ext)s" ] number
downloaded files starting from 00000
-t, --title [deprecated] use title in file name
(default)
-l, --literal [deprecated] alias of --title -l, --literal [deprecated] alias of --title
-w, --no-overwrites do not overwrite files -w, --no-overwrites do not overwrite files
-c, --continue force resume of partially downloaded files. -c, --continue force resume of partially downloaded files. By default, youtube-dl will resume downloads if possible.
By default, youtube-dl will resume --no-continue do not resume partially downloaded files (restart from beginning)
downloads if possible. --no-part do not use .part files - write directly into output file
--no-continue do not resume partially downloaded files --no-mtime do not use the Last-modified header to set the file modification time
(restart from beginning) --write-description write video description to a .description file
--no-part do not use .part files - write directly
into output file
--no-mtime do not use the Last-modified header to set
the file modification time
--write-description write video description to a .description
file
--write-info-json write video metadata to a .info.json file --write-info-json write video metadata to a .info.json file
--write-annotations write video annotations to a .annotation --write-annotations write video annotations to a .annotation file
file --load-info FILE json file containing the video information (created with the "--write-json" option)
--load-info FILE json file containing the video information --cookies FILE file to read cookies from and dump cookie jar in
(created with the "--write-json" option) --cache-dir DIR Location in the filesystem where youtube-dl can store some downloaded information permanently. By default $XDG_CACHE_HOME/youtube-dl
--cookies FILE file to read cookies from and dump cookie or ~/.cache/youtube-dl . At the moment, only YouTube player files (for videos with obfuscated signatures) are cached, but that may
jar in change.
--cache-dir DIR Location in the filesystem where youtube-dl
can store some downloaded information
permanently. By default $XDG_CACHE_HOME
/youtube-dl or ~/.cache/youtube-dl . At the
moment, only YouTube player files (for
videos with obfuscated signatures) are
cached, but that may change.
--no-cache-dir Disable filesystem caching --no-cache-dir Disable filesystem caching
--rm-cache-dir Delete all filesystem cache files --rm-cache-dir Delete all filesystem cache files
## Thumbnail images: ## Thumbnail images:
--write-thumbnail write thumbnail image to disk --write-thumbnail write thumbnail image to disk
--write-all-thumbnails write all thumbnail image formats to disk --write-all-thumbnails write all thumbnail image formats to disk
--list-thumbnails Simulate and list all available thumbnail --list-thumbnails Simulate and list all available thumbnail formats
formats
## Verbosity / Simulation Options: ## Verbosity / Simulation Options:
-q, --quiet activates quiet mode -q, --quiet activates quiet mode
--no-warnings Ignore warnings --no-warnings Ignore warnings
-s, --simulate do not download the video and do not write -s, --simulate do not download the video and do not write anything to disk
anything to disk
--skip-download do not download the video --skip-download do not download the video
-g, --get-url simulate, quiet but print URL -g, --get-url simulate, quiet but print URL
-e, --get-title simulate, quiet but print title -e, --get-title simulate, quiet but print title
@ -261,155 +159,87 @@ which means you can modify it, redistribute it or use it however you like.
--get-duration simulate, quiet but print video length --get-duration simulate, quiet but print video length
--get-filename simulate, quiet but print output filename --get-filename simulate, quiet but print output filename
--get-format simulate, quiet but print output format --get-format simulate, quiet but print output format
-j, --dump-json simulate, quiet but print JSON information. -j, --dump-json simulate, quiet but print JSON information. See --output for a description of available keys.
See --output for a description of available -J, --dump-single-json simulate, quiet but print JSON information for each command-line argument. If the URL refers to a playlist, dump the whole playlist
keys. information in a single line.
-J, --dump-single-json simulate, quiet but print JSON information --print-json Be quiet and print the video information as JSON (video is still being downloaded).
for each command-line argument. If the URL
refers to a playlist, dump the whole
playlist information in a single line.
--print-json Be quiet and print the video information as
JSON (video is still being downloaded).
--newline output progress bar as new lines --newline output progress bar as new lines
--no-progress do not print progress bar --no-progress do not print progress bar
--console-title display progress in console titlebar --console-title display progress in console titlebar
-v, --verbose print various debugging information -v, --verbose print various debugging information
--dump-intermediate-pages print downloaded pages to debug problems --dump-pages print downloaded pages to debug problems (very verbose)
(very verbose) --write-pages Write downloaded intermediary pages to files in the current directory to debug problems
--write-pages Write downloaded intermediary pages to
files in the current directory to debug
problems
--print-traffic Display sent and read HTTP traffic --print-traffic Display sent and read HTTP traffic
-C, --call-home Contact the youtube-dl server for -C, --call-home Contact the youtube-dl server for debugging.
debugging. --no-call-home Do NOT contact the youtube-dl server for debugging.
--no-call-home Do NOT contact the youtube-dl server for
debugging.
## Workarounds: ## Workarounds:
--encoding ENCODING Force the specified encoding (experimental) --encoding ENCODING Force the specified encoding (experimental)
--no-check-certificate Suppress HTTPS certificate validation. --no-check-certificate Suppress HTTPS certificate validation.
--prefer-insecure Use an unencrypted connection to retrieve --prefer-insecure Use an unencrypted connection to retrieve information about the video. (Currently supported only for YouTube)
information about the video. (Currently
supported only for YouTube)
--user-agent UA specify a custom user agent --user-agent UA specify a custom user agent
--referer URL specify a custom referer, use if the video --referer URL specify a custom referer, use if the video access is restricted to one domain
access is restricted to one domain --add-header FIELD:VALUE specify a custom HTTP header and its value, separated by a colon ':'. You can use this option multiple times
--add-header FIELD:VALUE specify a custom HTTP header and its value, --bidi-workaround Work around terminals that lack bidirectional text support. Requires bidiv or fribidi executable in PATH
separated by a colon ':'. You can use this --sleep-interval SECONDS Number of seconds to sleep before each download.
option multiple times
--bidi-workaround Work around terminals that lack
bidirectional text support. Requires bidiv
or fribidi executable in PATH
--sleep-interval SECONDS Number of seconds to sleep before each
download.
## Video Format Options: ## Video Format Options:
-f, --format FORMAT video format code, specify the order of -f, --format FORMAT video format code, specify the order of preference using slashes, as in -f 22/17/18 . Instead of format codes, you can select by
preference using slashes, as in -f 22/17/18 extension for the extensions aac, m4a, mp3, mp4, ogg, wav, webm. You can also use the special names "best", "bestvideo", "bestaudio",
. Instead of format codes, you can select "worst". You can filter the video results by putting a condition in brackets, as in -f "best[height=720]" (or -f "[filesize>10M]").
by extension for the extensions aac, m4a, This works for filesize, height, width, tbr, abr, vbr, asr, and fps and the comparisons <, <=, >, >=, =, != and for ext, acodec,
mp3, mp4, ogg, wav, webm. You can also use vcodec, container, and protocol and the comparisons =, != . Formats for which the value is not known are excluded unless you put a
the special names "best", "bestvideo", question mark (?) after the operator. You can combine format filters, so -f "[height <=? 720][tbr>500]" selects up to 720p videos
"bestaudio", "worst". You can filter the (or videos where the height is not known) with a bitrate of at least 500 KBit/s. By default, youtube-dl will pick the best quality.
video results by putting a condition in Use commas to download multiple audio formats, such as -f 136/137/mp4/bestvideo,140/m4a/bestaudio. You can merge the video and audio
brackets, as in -f "best[height=720]" (or of two formats into a single file using -f <video-format>+<audio-format> (requires ffmpeg or avconv), for example -f
-f "[filesize>10M]"). This works for
filesize, height, width, tbr, abr, vbr,
asr, and fps and the comparisons <, <=, >,
>=, =, != and for ext, acodec, vcodec,
container, and protocol and the comparisons
=, != . Formats for which the value is not
known are excluded unless you put a
question mark (?) after the operator. You
can combine format filters, so -f "[height
<=? 720][tbr>500]" selects up to 720p
videos (or videos where the height is not
known) with a bitrate of at least 500
KBit/s. By default, youtube-dl will pick
the best quality. Use commas to download
multiple audio formats, such as -f
136/137/mp4/bestvideo,140/m4a/bestaudio.
You can merge the video and audio of two
formats into a single file using -f <video-
format>+<audio-format> (requires ffmpeg or
avconv), for example -f
bestvideo+bestaudio. bestvideo+bestaudio.
--all-formats download all available video formats --all-formats download all available video formats
--prefer-free-formats prefer free video formats unless a specific --prefer-free-formats prefer free video formats unless a specific one is requested
one is requested
--max-quality FORMAT highest quality format to download --max-quality FORMAT highest quality format to download
-F, --list-formats list all available formats -F, --list-formats list all available formats
--youtube-skip-dash-manifest Do not download the DASH manifest on --youtube-skip-dash-manifest Do not download the DASH manifest on YouTube videos
YouTube videos --merge-output-format FORMAT If a merge is required (e.g. bestvideo+bestaudio), output to given container format. One of mkv, mp4, ogg, webm, flv.Ignored if no
--merge-output-format FORMAT If a merge is required (e.g. merge is required
bestvideo+bestaudio), output to given
container format. One of mkv, mp4, ogg,
webm, flv.Ignored if no merge is required
## Subtitle Options: ## Subtitle Options:
--write-sub write subtitle file --write-sub write subtitle file
--write-auto-sub write automatic subtitle file (youtube --write-auto-sub write automatic subtitle file (youtube only)
only) --all-subs downloads all the available subtitles of the video
--all-subs downloads all the available subtitles of
the video
--list-subs lists all available subtitles for the video --list-subs lists all available subtitles for the video
--sub-format FORMAT subtitle format, accepts formats --sub-format FORMAT subtitle format, accepts formats preference, for example: "ass/srt/best"
preference, for example: "ass/srt/best" --sub-lang LANGS languages of the subtitles to download (optional) separated by commas, use IETF language tags like 'en,pt'
--sub-lang LANGS languages of the subtitles to download
(optional) separated by commas, use IETF
language tags like 'en,pt'
## Authentication Options: ## Authentication Options:
-u, --username USERNAME login with this account ID -u, --username USERNAME login with this account ID
-p, --password PASSWORD account password. If this option is left -p, --password PASSWORD account password. If this option is left out, youtube-dl will ask interactively.
out, youtube-dl will ask interactively.
-2, --twofactor TWOFACTOR two-factor auth code -2, --twofactor TWOFACTOR two-factor auth code
-n, --netrc use .netrc authentication data -n, --netrc use .netrc authentication data
--video-password PASSWORD video password (vimeo, smotri) --video-password PASSWORD video password (vimeo, smotri)
## Post-processing Options: ## Post-processing Options:
-x, --extract-audio convert video files to audio-only files -x, --extract-audio convert video files to audio-only files (requires ffmpeg or avconv and ffprobe or avprobe)
(requires ffmpeg or avconv and ffprobe or --audio-format FORMAT "best", "aac", "vorbis", "mp3", "m4a", "opus", or "wav"; "best" by default
avprobe) --audio-quality QUALITY ffmpeg/avconv audio quality specification, insert a value between 0 (better) and 9 (worse) for VBR or a specific bitrate like 128K
--audio-format FORMAT "best", "aac", "vorbis", "mp3", "m4a", (default 5)
"opus", or "wav"; "best" by default --recode-video FORMAT Encode the video to another format if necessary (currently supported: mp4|flv|ogg|webm|mkv)
--audio-quality QUALITY ffmpeg/avconv audio quality specification, -k, --keep-video keeps the video file on disk after the post-processing; the video is erased by default
insert a value between 0 (better) and 9 --no-post-overwrites do not overwrite post-processed files; the post-processed files are overwritten by default
(worse) for VBR or a specific bitrate like --embed-subs embed subtitles in the video (only for mp4 videos)
128K (default 5)
--recode-video FORMAT Encode the video to another format if
necessary (currently supported:
mp4|flv|ogg|webm|mkv)
-k, --keep-video keeps the video file on disk after the
post-processing; the video is erased by
default
--no-post-overwrites do not overwrite post-processed files; the
post-processed files are overwritten by
default
--embed-subs embed subtitles in the video (only for mp4
videos)
--embed-thumbnail embed thumbnail in the audio as cover art --embed-thumbnail embed thumbnail in the audio as cover art
--add-metadata write metadata to the video file --add-metadata write metadata to the video file
--xattrs write metadata to the video file's xattrs --metadata-from-title FORMAT parse additional metadata like song title / artist from the video title. The format syntax is the same as --output, the parsed
(using dublin core and xdg standards) parameters replace existing values. Additional templates: %(album), %(artist). Example: --metadata-from-title "%(artist)s -
--fixup POLICY Automatically correct known faults of the %(title)s" matches a title like "Coldplay - Paradise"
file. One of never (do nothing), warn (only --xattrs write metadata to the video file's xattrs (using dublin core and xdg standards)
emit a warning), detect_or_warn(the --fixup POLICY Automatically correct known faults of the file. One of never (do nothing), warn (only emit a warning), detect_or_warn(the default;
default; fix file if we can, warn fix file if we can, warn otherwise)
otherwise) --prefer-avconv Prefer avconv over ffmpeg for running the postprocessors (default)
--prefer-avconv Prefer avconv over ffmpeg for running the --prefer-ffmpeg Prefer ffmpeg over avconv for running the postprocessors
postprocessors (default) --ffmpeg-location PATH Location of the ffmpeg/avconv binary; either the path to the binary or its containing directory.
--prefer-ffmpeg Prefer ffmpeg over avconv for running the --exec CMD Execute a command on the file after downloading, similar to find's -exec syntax. Example: --exec 'adb push {} /sdcard/Music/ && rm
postprocessors {}'
--ffmpeg-location PATH Location of the ffmpeg/avconv binary; --convert-subtitles FORMAT Convert the subtitles to other format (currently supported: srt|ass|vtt)
either the path to the binary or its
containing directory.
--exec CMD Execute a command on the file after
downloading, similar to find's -exec
syntax. Example: --exec 'adb push {}
/sdcard/Music/ && rm {}'
--convert-subtitles FORMAT Convert the subtitles to other format
(currently supported: srt|ass|vtt)
# CONFIGURATION # CONFIGURATION
@ -577,6 +407,18 @@ A note on the service that they don't host the infringing content, but just link
Support requests for services that **do** purchase the rights to distribute their content are perfectly fine though. If in doubt, you can simply include a source that mentions the legitimate purchase of content. Support requests for services that **do** purchase the rights to distribute their content are perfectly fine though. If in doubt, you can simply include a source that mentions the legitimate purchase of content.
### How can I speed up work on my issue?
(Also known as: Help, my important issue not being solved!) The youtube-dl core developer team is quite small. While we do our best to solve as many issues as possible, sometimes that can take quite a while. To speed up your issue, here's what you can do:
First of all, please do report the issue [at our issue tracker](https://yt-dl.org/bugs). That allows us to coordinate all efforts by users and developers, and serves as a unified point. Unfortunately, the youtube-dl project has grown too large to use personal email as an effective communication channel.
Please read the [bug reporting instructions](#bugs) below. A lot of bugs lack all the necessary information. If you can, offer proxy, VPN, or shell access to the youtube-dl developers. If you are able to, test the issue from multiple computers in multiple countries to exclude local censorship or misconfiguration issues.
If nobody is interested in solving your issue, you are welcome to take matters into your own hands and submit a pull request (or coerce/pay somebody else to do so).
Feel free to bump the issue from time to time by writing a small comment ("Issue is still present in youtube-dl version ...from France, but fixed from Belgium"), but please not more than once a month. Please do not declare your issue as `important` or `urgent`.
### How can I detect whether a given URL is supported by youtube-dl? ### How can I detect whether a given URL is supported by youtube-dl?
For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from http://example.com/video/1234567 to http://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug. For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from http://example.com/video/1234567 to http://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug.
@ -676,6 +518,7 @@ youtube-dl makes the best effort to be a good command-line program, and thus sho
From a Python program, you can embed youtube-dl in a more powerful fashion, like this: From a Python program, you can embed youtube-dl in a more powerful fashion, like this:
```python ```python
from __future__ import unicode_literals
import youtube_dl import youtube_dl
ydl_opts = {} ydl_opts = {}
@ -688,6 +531,7 @@ Most likely, you'll want to use various options. For a list of what can be done,
Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file: Here's a more complete example of a program that outputs only errors (and a short message after the download is finished), and downloads/converts the video to an mp3 file:
```python ```python
from __future__ import unicode_literals
import youtube_dl import youtube_dl
@ -745,7 +589,9 @@ If your report is shorter than two lines, it is almost certainly missing some of
For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the -v flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information. For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the -v flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information.
Site support requests **must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL. If your server has multiple IPs or you suspect censorship, adding --call-home may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like http://www.youtube.com/watch?v=BaW_jenozKc . There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. http://www.youtube.com/ ) is *not* an example URL.
### Are you using the latest version? ### Are you using the latest version?

13
docs/supportedsites.md
View File

@ -47,6 +47,7 @@
- **Bandcamp** - **Bandcamp**
- **Bandcamp:album** - **Bandcamp:album**
- **bbc.co.uk**: BBC iPlayer - **bbc.co.uk**: BBC iPlayer
- **BeatportPro**
- **Beeg** - **Beeg**
- **BehindKink** - **BehindKink**
- **Bet** - **Bet**
@ -117,6 +118,7 @@
- **DRTV** - **DRTV**
- **Dump** - **Dump**
- **dvtv**: http://video.aktualne.cz/ - **dvtv**: http://video.aktualne.cz/
- **EaglePlatform**
- **EbaumsWorld** - **EbaumsWorld**
- **EchoMsk** - **EchoMsk**
- **eHow** - **eHow**
@ -144,6 +146,7 @@
- **Firstpost** - **Firstpost**
- **Flickr** - **Flickr**
- **Folketinget**: Folketinget (ft.dk; Danish parliament) - **Folketinget**: Folketinget (ft.dk; Danish parliament)
- **FootyRoom**
- **Foxgay** - **Foxgay**
- **FoxNews** - **FoxNews**
- **france2.fr:generation-quoi** - **france2.fr:generation-quoi**
@ -161,6 +164,7 @@
- **GameSpot** - **GameSpot**
- **GameStar** - **GameStar**
- **Gametrailers** - **Gametrailers**
- **Gazeta**
- **GDCVault** - **GDCVault**
- **generic**: Generic downloader that works on some sites - **generic**: Generic downloader that works on some sites
- **GiantBomb** - **GiantBomb**
@ -211,6 +215,7 @@
- **jpopsuki.tv** - **jpopsuki.tv**
- **Jukebox** - **Jukebox**
- **Kaltura** - **Kaltura**
- **KanalPlay**: Kanal 5/9/11 Play
- **Kankan** - **Kankan**
- **Karaoketv** - **Karaoketv**
- **keek** - **keek**
@ -315,6 +320,7 @@
- **Ooyala** - **Ooyala**
- **OpenFilm** - **OpenFilm**
- **orf:fm4**: radio FM4 - **orf:fm4**: radio FM4
- **orf:iptv**: iptv.ORF.at
- **orf:oe1**: Radio Österreich 1 - **orf:oe1**: Radio Österreich 1
- **orf:tvthek**: ORF TVthek - **orf:tvthek**: ORF TVthek
- **parliamentlive.tv**: UK parliament videos - **parliamentlive.tv**: UK parliament videos
@ -322,10 +328,12 @@
- **PBS** - **PBS**
- **Phoenix** - **Phoenix**
- **Photobucket** - **Photobucket**
- **Pladform**
- **PlanetaPlay** - **PlanetaPlay**
- **play.fm** - **play.fm**
- **played.to** - **played.to**
- **Playvid** - **Playvid**
- **Playwire**
- **plus.google**: Google Plus - **plus.google**: Google Plus
- **pluzz.francetv.fr** - **pluzz.francetv.fr**
- **podomatic** - **podomatic**
@ -409,6 +417,7 @@
- **SportBox** - **SportBox**
- **SportDeutschland** - **SportDeutschland**
- **SRMediathek**: Saarländischer Rundfunk - **SRMediathek**: Saarländischer Rundfunk
- **SSA**
- **stanfordoc**: Stanford Open ClassRoom - **stanfordoc**: Stanford Open ClassRoom
- **Steam** - **Steam**
- **streamcloud.eu** - **streamcloud.eu**
@ -505,6 +514,7 @@
- **Vidzi** - **Vidzi**
- **vier** - **vier**
- **vier:videos** - **vier:videos**
- **Viewster**
- **viki** - **viki**
- **vimeo** - **vimeo**
- **vimeo:album** - **vimeo:album**
@ -551,6 +561,9 @@
- **XXXYMovies** - **XXXYMovies**
- **Yahoo**: Yahoo screen and movies - **Yahoo**: Yahoo screen and movies
- **Yam** - **Yam**
- **yandexmusic:album**: Яндекс.Музыка - Альбом
- **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
- **yandexmusic:track**: Яндекс.Музыка - Трек
- **YesJapan** - **YesJapan**
- **Ynet** - **Ynet**
- **YouJizz** - **YouJizz**

46
test/test_YoutubeDL.py
View File

@ -15,6 +15,8 @@ from youtube_dl import YoutubeDL
from youtube_dl.extractor import YoutubeIE from youtube_dl.extractor import YoutubeIE
from youtube_dl.postprocessor.common import PostProcessor from youtube_dl.postprocessor.common import PostProcessor
TEST_URL = 'http://localhost/sample.mp4'
class YDL(FakeYDL): class YDL(FakeYDL):
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
@ -46,8 +48,8 @@ class TestFormatSelection(unittest.TestCase):
ydl = YDL() ydl = YDL()
ydl.params['prefer_free_formats'] = True ydl.params['prefer_free_formats'] = True
formats = [ formats = [
{'ext': 'webm', 'height': 460, 'url': 'x'}, {'ext': 'webm', 'height': 460, 'url': TEST_URL},
{'ext': 'mp4', 'height': 460, 'url': 'y'}, {'ext': 'mp4', 'height': 460, 'url': TEST_URL},
] ]
info_dict = _make_result(formats) info_dict = _make_result(formats)
yie = YoutubeIE(ydl) yie = YoutubeIE(ydl)
@ -60,8 +62,8 @@ class TestFormatSelection(unittest.TestCase):
ydl = YDL() ydl = YDL()
ydl.params['prefer_free_formats'] = True ydl.params['prefer_free_formats'] = True
formats = [ formats = [
{'ext': 'webm', 'height': 720, 'url': 'a'}, {'ext': 'webm', 'height': 720, 'url': TEST_URL},
{'ext': 'mp4', 'height': 1080, 'url': 'b'}, {'ext': 'mp4', 'height': 1080, 'url': TEST_URL},
] ]
info_dict['formats'] = formats info_dict['formats'] = formats
yie = YoutubeIE(ydl) yie = YoutubeIE(ydl)
@ -74,9 +76,9 @@ class TestFormatSelection(unittest.TestCase):
ydl = YDL() ydl = YDL()
ydl.params['prefer_free_formats'] = False ydl.params['prefer_free_formats'] = False
formats = [ formats = [
{'ext': 'webm', 'height': 720, 'url': '_'}, {'ext': 'webm', 'height': 720, 'url': TEST_URL},
{'ext': 'mp4', 'height': 720, 'url': '_'}, {'ext': 'mp4', 'height': 720, 'url': TEST_URL},
{'ext': 'flv', 'height': 720, 'url': '_'}, {'ext': 'flv', 'height': 720, 'url': TEST_URL},
] ]
info_dict['formats'] = formats info_dict['formats'] = formats
yie = YoutubeIE(ydl) yie = YoutubeIE(ydl)
@ -88,8 +90,8 @@ class TestFormatSelection(unittest.TestCase):
ydl = YDL() ydl = YDL()
ydl.params['prefer_free_formats'] = False ydl.params['prefer_free_formats'] = False
formats = [ formats = [
{'ext': 'flv', 'height': 720, 'url': '_'}, {'ext': 'flv', 'height': 720, 'url': TEST_URL},
{'ext': 'webm', 'height': 720, 'url': '_'}, {'ext': 'webm', 'height': 720, 'url': TEST_URL},
] ]
info_dict['formats'] = formats info_dict['formats'] = formats
yie = YoutubeIE(ydl) yie = YoutubeIE(ydl)
@ -133,10 +135,10 @@ class TestFormatSelection(unittest.TestCase):
def test_format_selection(self): def test_format_selection(self):
formats = [ formats = [
{'format_id': '35', 'ext': 'mp4', 'preference': 1, 'url': '_'}, {'format_id': '35', 'ext': 'mp4', 'preference': 1, 'url': TEST_URL},
{'format_id': '45', 'ext': 'webm', 'preference': 2, 'url': '_'}, {'format_id': '45', 'ext': 'webm', 'preference': 2, 'url': TEST_URL},
{'format_id': '47', 'ext': 'webm', 'preference': 3, 'url': '_'}, {'format_id': '47', 'ext': 'webm', 'preference': 3, 'url': TEST_URL},
{'format_id': '2', 'ext': 'flv', 'preference': 4, 'url': '_'}, {'format_id': '2', 'ext': 'flv', 'preference': 4, 'url': TEST_URL},
] ]
info_dict = _make_result(formats) info_dict = _make_result(formats)
@ -167,10 +169,10 @@ class TestFormatSelection(unittest.TestCase):
def test_format_selection_audio(self): def test_format_selection_audio(self):
formats = [ formats = [
{'format_id': 'audio-low', 'ext': 'webm', 'preference': 1, 'vcodec': 'none', 'url': '_'}, {'format_id': 'audio-low', 'ext': 'webm', 'preference': 1, 'vcodec': 'none', 'url': TEST_URL},
{'format_id': 'audio-mid', 'ext': 'webm', 'preference': 2, 'vcodec': 'none', 'url': '_'}, {'format_id': 'audio-mid', 'ext': 'webm', 'preference': 2, 'vcodec': 'none', 'url': TEST_URL},
{'format_id': 'audio-high', 'ext': 'flv', 'preference': 3, 'vcodec': 'none', 'url': '_'}, {'format_id': 'audio-high', 'ext': 'flv', 'preference': 3, 'vcodec': 'none', 'url': TEST_URL},
{'format_id': 'vid', 'ext': 'mp4', 'preference': 4, 'url': '_'}, {'format_id': 'vid', 'ext': 'mp4', 'preference': 4, 'url': TEST_URL},
] ]
info_dict = _make_result(formats) info_dict = _make_result(formats)
@ -185,8 +187,8 @@ class TestFormatSelection(unittest.TestCase):
self.assertEqual(downloaded['format_id'], 'audio-low') self.assertEqual(downloaded['format_id'], 'audio-low')
formats = [ formats = [
{'format_id': 'vid-low', 'ext': 'mp4', 'preference': 1, 'url': '_'}, {'format_id': 'vid-low', 'ext': 'mp4', 'preference': 1, 'url': TEST_URL},
{'format_id': 'vid-high', 'ext': 'mp4', 'preference': 2, 'url': '_'}, {'format_id': 'vid-high', 'ext': 'mp4', 'preference': 2, 'url': TEST_URL},
] ]
info_dict = _make_result(formats) info_dict = _make_result(formats)
@ -228,9 +230,9 @@ class TestFormatSelection(unittest.TestCase):
def test_format_selection_video(self): def test_format_selection_video(self):
formats = [ formats = [
{'format_id': 'dash-video-low', 'ext': 'mp4', 'preference': 1, 'acodec': 'none', 'url': '_'}, {'format_id': 'dash-video-low', 'ext': 'mp4', 'preference': 1, 'acodec': 'none', 'url': TEST_URL},
{'format_id': 'dash-video-high', 'ext': 'mp4', 'preference': 2, 'acodec': 'none', 'url': '_'}, {'format_id': 'dash-video-high', 'ext': 'mp4', 'preference': 2, 'acodec': 'none', 'url': TEST_URL},
{'format_id': 'vid', 'ext': 'mp4', 'preference': 3, 'url': '_'}, {'format_id': 'vid', 'ext': 'mp4', 'preference': 3, 'url': TEST_URL},
] ]
info_dict = _make_result(formats) info_dict = _make_result(formats)

10
test/test_all_urls.py
View File

@ -104,11 +104,11 @@ class TestAllURLsMatching(unittest.TestCase):
self.assertMatch(':tds', ['ComedyCentralShows']) self.assertMatch(':tds', ['ComedyCentralShows'])
def test_vimeo_matching(self): def test_vimeo_matching(self):
self.assertMatch('http://vimeo.com/channels/tributes', ['vimeo:channel']) self.assertMatch('https://vimeo.com/channels/tributes', ['vimeo:channel'])
self.assertMatch('http://vimeo.com/channels/31259', ['vimeo:channel']) self.assertMatch('https://vimeo.com/channels/31259', ['vimeo:channel'])
self.assertMatch('http://vimeo.com/channels/31259/53576664', ['vimeo']) self.assertMatch('https://vimeo.com/channels/31259/53576664', ['vimeo'])
self.assertMatch('http://vimeo.com/user7108434', ['vimeo:user']) self.assertMatch('https://vimeo.com/user7108434', ['vimeo:user'])
self.assertMatch('http://vimeo.com/user7108434/videos', ['vimeo:user']) self.assertMatch('https://vimeo.com/user7108434/videos', ['vimeo:user'])
self.assertMatch('https://vimeo.com/user21297594/review/75524534/3c257a1b5d', ['vimeo:review']) self.assertMatch('https://vimeo.com/user21297594/review/75524534/3c257a1b5d', ['vimeo:review'])
# https://github.com/rg3/youtube-dl/issues/1930 # https://github.com/rg3/youtube-dl/issues/1930

26
test/test_netrc.py Normal file
View File

@ -0,0 +1,26 @@
# coding: utf-8
from __future__ import unicode_literals
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from youtube_dl.extractor import (
gen_extractors,
)
class TestNetRc(unittest.TestCase):
def test_netrc_present(self):
for ie in gen_extractors():
if not hasattr(ie, '_login'):
continue
self.assertTrue(
hasattr(ie, '_NETRC_MACHINE'),
'Extractor %s supports login, but is missing a _NETRC_MACHINE property' % ie.IE_NAME)
if __name__ == '__main__':
unittest.main()

17
test/test_postprocessors.py Normal file
View File

@ -0,0 +1,17 @@
#!/usr/bin/env python
from __future__ import unicode_literals
# Allow direct execution
import os
import sys
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from youtube_dl.postprocessor import MetadataFromTitlePP
class TestMetadataFromTitle(unittest.TestCase):
def test_format_to_regex(self):
pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
self.assertEqual(pp._titleregex, '(?P<title>.+)\ \-\ (?P<artist>.+)')

13
test/test_subtitles.py
View File

@ -26,6 +26,7 @@ from youtube_dl.extractor import (
VikiIE, VikiIE,
ThePlatformIE, ThePlatformIE,
RTVEALaCartaIE, RTVEALaCartaIE,
FunnyOrDieIE,
) )
@ -320,5 +321,17 @@ class TestRtveSubtitles(BaseTestSubtitles):
self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca') self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca')
class TestFunnyOrDieSubtitles(BaseTestSubtitles):
url = 'http://www.funnyordie.com/videos/224829ff6d/judd-apatow-will-direct-your-vine'
IE = FunnyOrDieIE
def test_allsubtitles(self):
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(set(subtitles.keys()), set(['en']))
self.assertEqual(md5(subtitles['en']), 'c5593c193eacd353596c11c2d4f9ecc4')
if __name__ == '__main__': if __name__ == '__main__':
unittest.main() unittest.main()

11
test/test_unicode_literals.py
View File

@ -17,13 +17,22 @@ IGNORED_FILES = [
'buildserver.py', 'buildserver.py',
] ]
IGNORED_DIRS = [
'.git',
'.tox',
]
from test.helper import assertRegexpMatches from test.helper import assertRegexpMatches
class TestUnicodeLiterals(unittest.TestCase): class TestUnicodeLiterals(unittest.TestCase):
def test_all_files(self): def test_all_files(self):
for dirpath, _, filenames in os.walk(rootDir): for dirpath, dirnames, filenames in os.walk(rootDir):
for ignore_dir in IGNORED_DIRS:
if ignore_dir in dirnames:
# If we remove the directory from dirnames os.walk won't
# recurse into it
dirnames.remove(ignore_dir)
for basename in filenames: for basename in filenames:
if not basename.endswith('.py'): if not basename.endswith('.py'):
continue continue

40
test/test_utils.py
View File

@ -38,6 +38,7 @@ from youtube_dl.utils import (
parse_iso8601, parse_iso8601,
read_batch_urls, read_batch_urls,
sanitize_filename, sanitize_filename,
sanitize_path,
shell_quote, shell_quote,
smuggle_url, smuggle_url,
str_to_int, str_to_int,
@ -85,8 +86,11 @@ class TestUtil(unittest.TestCase):
self.assertEqual( self.assertEqual(
sanitize_filename('New World record at 0:12:34'), sanitize_filename('New World record at 0:12:34'),
'New World record at 0_12_34') 'New World record at 0_12_34')
self.assertEqual(sanitize_filename('--gasdgf'), '_-gasdgf') self.assertEqual(sanitize_filename('--gasdgf'), '_-gasdgf')
self.assertEqual(sanitize_filename('--gasdgf', is_id=True), '--gasdgf') self.assertEqual(sanitize_filename('--gasdgf', is_id=True), '--gasdgf')
self.assertEqual(sanitize_filename('.gasdgf'), 'gasdgf')
self.assertEqual(sanitize_filename('.gasdgf', is_id=True), '.gasdgf')
forbidden = '"\0\\/' forbidden = '"\0\\/'
for fc in forbidden: for fc in forbidden:
@ -128,6 +132,42 @@ class TestUtil(unittest.TestCase):
self.assertEqual(sanitize_filename('_BD_eEpuzXw', is_id=True), '_BD_eEpuzXw') self.assertEqual(sanitize_filename('_BD_eEpuzXw', is_id=True), '_BD_eEpuzXw')
self.assertEqual(sanitize_filename('N0Y__7-UOdI', is_id=True), 'N0Y__7-UOdI') self.assertEqual(sanitize_filename('N0Y__7-UOdI', is_id=True), 'N0Y__7-UOdI')
def test_sanitize_path(self):
if sys.platform != 'win32':
return
self.assertEqual(sanitize_path('abc'), 'abc')
self.assertEqual(sanitize_path('abc/def'), 'abc\\def')
self.assertEqual(sanitize_path('abc\\def'), 'abc\\def')
self.assertEqual(sanitize_path('abc|def'), 'abc#def')
self.assertEqual(sanitize_path('<>:"|?*'), '#######')
self.assertEqual(sanitize_path('C:/abc/def'), 'C:\\abc\\def')
self.assertEqual(sanitize_path('C?:/abc/def'), 'C##\\abc\\def')
self.assertEqual(sanitize_path('\\\\?\\UNC\\ComputerName\\abc'), '\\\\?\\UNC\\ComputerName\\abc')
self.assertEqual(sanitize_path('\\\\?\\UNC/ComputerName/abc'), '\\\\?\\UNC\\ComputerName\\abc')
self.assertEqual(sanitize_path('\\\\?\\C:\\abc'), '\\\\?\\C:\\abc')
self.assertEqual(sanitize_path('\\\\?\\C:/abc'), '\\\\?\\C:\\abc')
self.assertEqual(sanitize_path('\\\\?\\C:\\ab?c\\de:f'), '\\\\?\\C:\\ab#c\\de#f')
self.assertEqual(sanitize_path('\\\\?\\C:\\abc'), '\\\\?\\C:\\abc')
self.assertEqual(
sanitize_path('youtube/%(uploader)s/%(autonumber)s-%(title)s-%(upload_date)s.%(ext)s'),
'youtube\\%(uploader)s\\%(autonumber)s-%(title)s-%(upload_date)s.%(ext)s')
self.assertEqual(
sanitize_path('youtube/TheWreckingYard ./00001-Not bad, Especially for Free! (1987 Yamaha 700)-20141116.mp4.part'),
'youtube\\TheWreckingYard #\\00001-Not bad, Especially for Free! (1987 Yamaha 700)-20141116.mp4.part')
self.assertEqual(sanitize_path('abc/def...'), 'abc\\def..#')
self.assertEqual(sanitize_path('abc.../def'), 'abc..#\\def')
self.assertEqual(sanitize_path('abc.../def...'), 'abc..#\\def..#')
self.assertEqual(sanitize_path('../abc'), '..\\abc')
self.assertEqual(sanitize_path('../../abc'), '..\\..\\abc')
self.assertEqual(sanitize_path('./abc'), 'abc')
self.assertEqual(sanitize_path('./../abc'), '..\\abc')
def test_ordered_set(self): def test_ordered_set(self):
self.assertEqual(orderedSet([1, 1, 2, 3, 4, 4, 5, 6, 7, 3, 5]), [1, 2, 3, 4, 5, 6, 7]) self.assertEqual(orderedSet([1, 1, 2, 3, 4, 4, 5, 6, 7, 3, 5]), [1, 2, 3, 4, 5, 6, 7])
self.assertEqual(orderedSet([]), []) self.assertEqual(orderedSet([]), [])

7
tox.ini
View File

@ -1,8 +1,11 @@
[tox] [tox]
envlist = py26,py27,py33 envlist = py26,py27,py33,py34
[testenv] [testenv]
deps = deps =
nose nose
coverage coverage
commands = nosetests --verbose {posargs:test} # --with-coverage --cover-package=youtube_dl --cover-html defaultargs = test --exclude test_download.py --exclude test_age_restriction.py
--exclude test_subtitles.py --exclude test_write_annotations.py
--exclude test_youtube_lists.py
commands = nosetests --verbose {posargs:{[testenv]defaultargs}} # --with-coverage --cover-package=youtube_dl --cover-html
# test.test_download:TestDownload.test_NowVideo # test.test_download:TestDownload.test_NowVideo

27
youtube_dl/YoutubeDL.py
View File

@ -54,12 +54,14 @@ from .utils import (
MaxDownloadsReached, MaxDownloadsReached,
PagedList, PagedList,
parse_filesize, parse_filesize,
PerRequestProxyHandler,
PostProcessingError, PostProcessingError,
platform_name, platform_name,
preferredencoding, preferredencoding,
render_table, render_table,
SameFileError, SameFileError,
sanitize_filename, sanitize_filename,
sanitize_path,
std_headers, std_headers,
subtitles_filename, subtitles_filename,
build_part_filename, build_part_filename,
@ -184,6 +186,8 @@ class YoutubeDL(object):
prefer_insecure: Use HTTP instead of HTTPS to retrieve information. prefer_insecure: Use HTTP instead of HTTPS to retrieve information.
At the moment, this is only supported by YouTube. At the moment, this is only supported by YouTube.
proxy: URL of the proxy server to use proxy: URL of the proxy server to use
cn_verification_proxy: URL of the proxy to use for IP address verification
on Chinese sites. (Experimental)
socket_timeout: Time to wait for unresponsive hosts, in seconds socket_timeout: Time to wait for unresponsive hosts, in seconds
bidi_workaround: Work around buggy terminals without bidirectional text bidi_workaround: Work around buggy terminals without bidirectional text
support, using fridibi support, using fridibi
@ -250,10 +254,10 @@ class YoutubeDL(object):
hls_prefer_native: Use the native HLS downloader instead of ffmpeg/avconv. hls_prefer_native: Use the native HLS downloader instead of ffmpeg/avconv.
The following parameters are not used by YoutubeDL itself, they are used by The following parameters are not used by YoutubeDL itself, they are used by
the FileDownloader: the downloader (see youtube_dl/downloader/common.py):
nopart, updatetime, buffersize, ratelimit, min_filesize, max_filesize, test, nopart, updatetime, buffersize, ratelimit, min_filesize, max_filesize, test,
noresizebuffer, retries, continuedl, noprogress, consoletitle, noresizebuffer, retries, continuedl, noprogress, consoletitle,
xattr_set_filesize. xattr_set_filesize, external_downloader_args.
The following options are used by the post processors: The following options are used by the post processors:
prefer_ffmpeg: If True, use ffmpeg instead of avconv if both are available, prefer_ffmpeg: If True, use ffmpeg instead of avconv if both are available,
@ -320,6 +324,11 @@ class YoutubeDL(object):
'Set the LC_ALL environment variable to fix this.') 'Set the LC_ALL environment variable to fix this.')
self.params['restrictfilenames'] = True self.params['restrictfilenames'] = True
if isinstance(params.get('outtmpl'), bytes):
self.report_warning(
'Parameter outtmpl is bytes, but should be a unicode string. '
'Put from __future__ import unicode_literals at the top of your code file or consider switching to Python 3.x.')
if '%(stitle)s' in self.params.get('outtmpl', ''): if '%(stitle)s' in self.params.get('outtmpl', ''):
self.report_warning('%(stitle)s is deprecated. Use the %(title)s and the --restrict-filenames flag(which also secures %(uploader)s et al) instead.') self.report_warning('%(stitle)s is deprecated. Use the %(title)s and the --restrict-filenames flag(which also secures %(uploader)s et al) instead.')
@ -560,7 +569,7 @@ class YoutubeDL(object):
if v is not None) if v is not None)
template_dict = collections.defaultdict(lambda: 'NA', template_dict) template_dict = collections.defaultdict(lambda: 'NA', template_dict)
outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL) outtmpl = sanitize_path(self.params.get('outtmpl', DEFAULT_OUTTMPL))
tmpl = compat_expanduser(outtmpl) tmpl = compat_expanduser(outtmpl)
filename = tmpl % template_dict filename = tmpl % template_dict
# Temporary fix for #4787 # Temporary fix for #4787
@ -627,7 +636,7 @@ class YoutubeDL(object):
Returns a list with a dictionary for each video we find. Returns a list with a dictionary for each video we find.
If 'download', also downloads the videos. If 'download', also downloads the videos.
extra_info is a dict containing the extra values to add to each result extra_info is a dict containing the extra values to add to each result
''' '''
if ie_key: if ie_key:
ies = [self.get_info_extractor(ie_key)] ies = [self.get_info_extractor(ie_key)]
@ -1092,8 +1101,7 @@ class YoutubeDL(object):
if req_format is None: if req_format is None:
req_format = 'best' req_format = 'best'
formats_to_download = [] formats_to_download = []
# The -1 is for supporting YoutubeIE if req_format == 'all':
if req_format in ('-1', 'all'):
formats_to_download = formats formats_to_download = formats
else: else:
for rfstr in req_format.split(','): for rfstr in req_format.split(','):
@ -1268,7 +1276,7 @@ class YoutubeDL(object):
return return
try: try:
dn = os.path.dirname(encodeFilename(filename)) dn = os.path.dirname(sanitize_path(encodeFilename(filename)))
if dn and not os.path.exists(dn): if dn and not os.path.exists(dn):
os.makedirs(dn) os.makedirs(dn)
except (OSError, IOError) as err: except (OSError, IOError) as err:
@ -1791,13 +1799,14 @@ class YoutubeDL(object):
# Set HTTPS proxy to HTTP one if given (https://github.com/rg3/youtube-dl/issues/805) # Set HTTPS proxy to HTTP one if given (https://github.com/rg3/youtube-dl/issues/805)
if 'http' in proxies and 'https' not in proxies: if 'http' in proxies and 'https' not in proxies:
proxies['https'] = proxies['http'] proxies['https'] = proxies['http']
proxy_handler = compat_urllib_request.ProxyHandler(proxies) proxy_handler = PerRequestProxyHandler(proxies)
debuglevel = 1 if self.params.get('debug_printtraffic') else 0 debuglevel = 1 if self.params.get('debug_printtraffic') else 0
https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel) https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
ydlh = YoutubeDLHandler(self.params, debuglevel=debuglevel) ydlh = YoutubeDLHandler(self.params, debuglevel=debuglevel)
opener = compat_urllib_request.build_opener( opener = compat_urllib_request.build_opener(
https_handler, proxy_handler, cookie_processor, ydlh) proxy_handler, https_handler, cookie_processor, ydlh)
# Delete the default user-agent header, which would otherwise apply in # Delete the default user-agent header, which would otherwise apply in
# cases where our custom HTTP handler doesn't come into play # cases where our custom HTTP handler doesn't come into play
# (See https://github.com/rg3/youtube-dl/issues/1309 for details) # (See https://github.com/rg3/youtube-dl/issues/1309 for details)

11
youtube_dl/__init__.py
View File

@ -9,6 +9,7 @@ import codecs
import io import io
import os import os
import random import random
import shlex
import sys import sys
@ -212,6 +213,11 @@ def _real_main(argv=None):
# PostProcessors # PostProcessors
postprocessors = [] postprocessors = []
# Add the metadata pp first, the other pps will copy it # Add the metadata pp first, the other pps will copy it
if opts.metafromtitle:
postprocessors.append({
'key': 'MetadataFromTitle',
'titleformat': opts.metafromtitle
})
if opts.addmetadata: if opts.addmetadata:
postprocessors.append({'key': 'FFmpegMetadata'}) postprocessors.append({'key': 'FFmpegMetadata'})
if opts.extractaudio: if opts.extractaudio:
@ -257,6 +263,9 @@ def _real_main(argv=None):
xattr # Confuse flake8 xattr # Confuse flake8
except ImportError: except ImportError:
parser.error('setting filesize xattr requested but python-xattr is not available') parser.error('setting filesize xattr requested but python-xattr is not available')
external_downloader_args = None
if opts.external_downloader_args:
external_downloader_args = shlex.split(opts.external_downloader_args)
match_filter = ( match_filter = (
None if opts.match_filter is None None if opts.match_filter is None
else match_filter_func(opts.match_filter)) else match_filter_func(opts.match_filter))
@ -362,6 +371,8 @@ def _real_main(argv=None):
'no_color': opts.no_color, 'no_color': opts.no_color,
'ffmpeg_location': opts.ffmpeg_location, 'ffmpeg_location': opts.ffmpeg_location,
'hls_prefer_native': opts.hls_prefer_native, 'hls_prefer_native': opts.hls_prefer_native,
'external_downloader_args': external_downloader_args,
'cn_verification_proxy': opts.cn_verification_proxy,
} }
with YoutubeDL(ydl_opts) as ydl: with YoutubeDL(ydl_opts) as ydl:

2
youtube_dl/downloader/common.py
View File

@ -42,6 +42,8 @@ class FileDownloader(object):
max_filesize: Skip files larger than this size max_filesize: Skip files larger than this size
xattr_set_filesize: Set ytdl.filesize user xattribute with expected size. xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
(experimenatal) (experimenatal)
external_downloader_args: A list of additional command-line arguments for the
external downloader.
Subclasses of this one must re-define the real_download method. Subclasses of this one must re-define the real_download method.
""" """

15
youtube_dl/downloader/external.py
View File

@ -51,6 +51,13 @@ class ExternalFD(FileDownloader):
return [] return []
return [command_option, source_address] return [command_option, source_address]
def _configuration_args(self, default=[]):
ex_args = self.params.get('external_downloader_args')
if ex_args is None:
return default
assert isinstance(ex_args, list)
return ex_args
def _call_downloader(self, tmpfilename, info_dict): def _call_downloader(self, tmpfilename, info_dict):
""" Either overwrite this or implement _make_cmd """ """ Either overwrite this or implement _make_cmd """
cmd = self._make_cmd(tmpfilename, info_dict) cmd = self._make_cmd(tmpfilename, info_dict)
@ -79,6 +86,7 @@ class CurlFD(ExternalFD):
for key, val in info_dict['http_headers'].items(): for key, val in info_dict['http_headers'].items():
cmd += ['--header', '%s: %s' % (key, val)] cmd += ['--header', '%s: %s' % (key, val)]
cmd += self._source_address('--interface') cmd += self._source_address('--interface')
cmd += self._configuration_args()
cmd += ['--', info_dict['url']] cmd += ['--', info_dict['url']]
return cmd return cmd
@ -89,15 +97,16 @@ class WgetFD(ExternalFD):
for key, val in info_dict['http_headers'].items(): for key, val in info_dict['http_headers'].items():
cmd += ['--header', '%s: %s' % (key, val)] cmd += ['--header', '%s: %s' % (key, val)]
cmd += self._source_address('--bind-address') cmd += self._source_address('--bind-address')
cmd += self._configuration_args()
cmd += ['--', info_dict['url']] cmd += ['--', info_dict['url']]
return cmd return cmd
class Aria2cFD(ExternalFD): class Aria2cFD(ExternalFD):
def _make_cmd(self, tmpfilename, info_dict): def _make_cmd(self, tmpfilename, info_dict):
cmd = [ cmd = [self.exe, '-c']
self.exe, '-c', cmd += self._configuration_args([
'--min-split-size', '1M', '--max-connection-per-server', '4'] '--min-split-size', '1M', '--max-connection-per-server', '4'])
dn = os.path.dirname(tmpfilename) dn = os.path.dirname(tmpfilename)
if dn: if dn:
cmd += ['--dir', dn] cmd += ['--dir', dn]

4
youtube_dl/downloader/f4m.py
View File

@ -281,7 +281,7 @@ class F4mFD(FileDownloader):
boot_info = self._get_bootstrap_from_url(bootstrap_url) boot_info = self._get_bootstrap_from_url(bootstrap_url)
else: else:
bootstrap_url = None bootstrap_url = None
bootstrap = base64.b64decode(node.text) bootstrap = base64.b64decode(node.text.encode('ascii'))
boot_info = read_bootstrap_info(bootstrap) boot_info = read_bootstrap_info(bootstrap)
return (boot_info, bootstrap_url) return (boot_info, bootstrap_url)
@ -308,7 +308,7 @@ class F4mFD(FileDownloader):
live = boot_info['live'] live = boot_info['live']
metadata_node = media.find(_add_ns('metadata')) metadata_node = media.find(_add_ns('metadata'))
if metadata_node is not None: if metadata_node is not None:
metadata = base64.b64decode(metadata_node.text) metadata = base64.b64decode(metadata_node.text.encode('ascii'))
else: else:
metadata = None metadata = None

8
youtube_dl/downloader/http.py
View File

@ -92,6 +92,8 @@ class HttpFD(FileDownloader):
self._hook_progress({ self._hook_progress({
'filename': filename, 'filename': filename,
'status': 'finished', 'status': 'finished',
'downloaded_bytes': resume_len,
'total_bytes': resume_len,
}) })
return True return True
else: else:
@ -218,12 +220,6 @@ class HttpFD(FileDownloader):
if tmpfilename != '-': if tmpfilename != '-':
stream.close() stream.close()
self._hook_progress({
'downloaded_bytes': byte_counter,
'total_bytes': data_len,
'tmpfilename': tmpfilename,
'status': 'error',
})
if data_len is not None and byte_counter != data_len: if data_len is not None and byte_counter != data_len:
raise ContentTooShortError(byte_counter, int(data_len)) raise ContentTooShortError(byte_counter, int(data_len))
self.try_rename(tmpfilename, filename) self.try_rename(tmpfilename, filename)

17
youtube_dl/extractor/__init__.py
View File

@ -37,6 +37,7 @@ from .bandcamp import BandcampIE, BandcampAlbumIE
from .bbccouk import BBCCoUkIE from .bbccouk import BBCCoUkIE
from .beeg import BeegIE from .beeg import BeegIE
from .behindkink import BehindKinkIE from .behindkink import BehindKinkIE
from .beatportpro import BeatportProIE
from .bet import BetIE from .bet import BetIE
from .bild import BildIE from .bild import BildIE
from .bilibili import BiliBiliIE from .bilibili import BiliBiliIE
@ -116,6 +117,7 @@ from .defense import DefenseGouvFrIE
from .discovery import DiscoveryIE from .discovery import DiscoveryIE
from .divxstage import DivxStageIE from .divxstage import DivxStageIE
from .dropbox import DropboxIE from .dropbox import DropboxIE
from .eagleplatform import EaglePlatformIE
from .ebaumsworld import EbaumsWorldIE from .ebaumsworld import EbaumsWorldIE
from .echomsk import EchoMskIE from .echomsk import EchoMskIE
from .ehow import EHowIE from .ehow import EHowIE
@ -150,6 +152,7 @@ from .fktv import (
) )
from .flickr import FlickrIE from .flickr import FlickrIE
from .folketinget import FolketingetIE from .folketinget import FolketingetIE
from .footyroom import FootyRoomIE
from .fourtube import FourTubeIE from .fourtube import FourTubeIE
from .foxgay import FoxgayIE from .foxgay import FoxgayIE
from .foxnews import FoxNewsIE from .foxnews import FoxNewsIE
@ -174,6 +177,7 @@ from .gameone import (
from .gamespot import GameSpotIE from .gamespot import GameSpotIE
from .gamestar import GameStarIE from .gamestar import GameStarIE
from .gametrailers import GametrailersIE from .gametrailers import GametrailersIE
from .gazeta import GazetaIE
from .gdcvault import GDCVaultIE from .gdcvault import GDCVaultIE
from .generic import GenericIE from .generic import GenericIE
from .giantbomb import GiantBombIE from .giantbomb import GiantBombIE
@ -228,6 +232,7 @@ from .jove import JoveIE
from .jukebox import JukeboxIE from .jukebox import JukeboxIE
from .jpopsukitv import JpopsukiIE from .jpopsukitv import JpopsukiIE
from .kaltura import KalturaIE from .kaltura import KalturaIE
from .kanalplay import KanalPlayIE
from .kankan import KankanIE from .kankan import KankanIE
from .karaoketv import KaraoketvIE from .karaoketv import KaraoketvIE
from .keezmovies import KeezMoviesIE from .keezmovies import KeezMoviesIE
@ -354,6 +359,7 @@ from .orf import (
ORFTVthekIE, ORFTVthekIE,
ORFOE1IE, ORFOE1IE,
ORFFM4IE, ORFFM4IE,
ORFIPTVIE,
) )
from .parliamentliveuk import ParliamentLiveUKIE from .parliamentliveuk import ParliamentLiveUKIE
from .patreon import PatreonIE from .patreon import PatreonIE
@ -361,9 +367,11 @@ from .pbs import PBSIE
from .phoenix import PhoenixIE from .phoenix import PhoenixIE
from .photobucket import PhotobucketIE from .photobucket import PhotobucketIE
from .planetaplay import PlanetaPlayIE from .planetaplay import PlanetaPlayIE
from .pladform import PladformIE
from .played import PlayedIE from .played import PlayedIE
from .playfm import PlayFMIE from .playfm import PlayFMIE
from .playvid import PlayvidIE from .playvid import PlayvidIE
from .playwire import PlaywireIE
from .podomatic import PodomaticIE from .podomatic import PodomaticIE
from .pornhd import PornHdIE from .pornhd import PornHdIE
from .pornhub import ( from .pornhub import (
@ -397,7 +405,7 @@ from .rtlnow import RTLnowIE
from .rtl2 import RTL2IE from .rtl2 import RTL2IE
from .rtp import RTPIE from .rtp import RTPIE
from .rts import RTSIE from .rts import RTSIE
from .rtve import RTVEALaCartaIE, RTVELiveIE from .rtve import RTVEALaCartaIE, RTVELiveIE, RTVEInfantilIE
from .ruhd import RUHDIE from .ruhd import RUHDIE
from .rutube import ( from .rutube import (
RutubeIE, RutubeIE,
@ -455,6 +463,7 @@ from .sport5 import Sport5IE
from .sportbox import SportBoxIE from .sportbox import SportBoxIE
from .sportdeutschland import SportDeutschlandIE from .sportdeutschland import SportDeutschlandIE
from .srmediathek import SRMediathekIE from .srmediathek import SRMediathekIE
from .ssa import SSAIE
from .stanfordoc import StanfordOpenClassroomIE from .stanfordoc import StanfordOpenClassroomIE
from .steam import SteamIE from .steam import SteamIE
from .streamcloud import StreamcloudIE from .streamcloud import StreamcloudIE
@ -550,6 +559,7 @@ from .videoweed import VideoWeedIE
from .vidme import VidmeIE from .vidme import VidmeIE
from .vidzi import VidziIE from .vidzi import VidziIE
from .vier import VierIE, VierVideosIE from .vier import VierIE, VierVideosIE
from .viewster import ViewsterIE
from .vimeo import ( from .vimeo import (
VimeoIE, VimeoIE,
VimeoAlbumIE, VimeoAlbumIE,
@ -606,6 +616,11 @@ from .yahoo import (
YahooSearchIE, YahooSearchIE,
) )
from .yam import YamIE from .yam import YamIE
from .yandexmusic import (
YandexMusicTrackIE,
YandexMusicAlbumIE,
YandexMusicPlaylistIE,
)
from .yesjapan import YesJapanIE from .yesjapan import YesJapanIE
from .ynet import YnetIE from .ynet import YnetIE
from .youjizz import YouJizzIE from .youjizz import YouJizzIE

44
youtube_dl/extractor/adultswim.py
View File

@ -2,13 +2,12 @@
from __future__ import unicode_literals from __future__ import unicode_literals
import re import re
import json
from .common import InfoExtractor from .common import InfoExtractor
from ..utils import ( from ..utils import (
ExtractorError, ExtractorError,
xpath_text,
float_or_none, float_or_none,
xpath_text,
) )
@ -60,6 +59,24 @@ class AdultSwimIE(InfoExtractor):
'title': 'American Dad - Putting Francine Out of Business', 'title': 'American Dad - Putting Francine Out of Business',
'description': 'Stan hatches a plan to get Francine out of the real estate business.Watch more American Dad on [adult swim].' 'description': 'Stan hatches a plan to get Francine out of the real estate business.Watch more American Dad on [adult swim].'
}, },
}, {
'url': 'http://www.adultswim.com/videos/tim-and-eric-awesome-show-great-job/dr-steve-brule-for-your-wine/',
'playlist': [
{
'md5': '3e346a2ab0087d687a05e1e7f3b3e529',
'info_dict': {
'id': 'sY3cMUR_TbuE4YmdjzbIcQ-0',
'ext': 'flv',
'title': 'Tim and Eric Awesome Show Great Job! - Dr. Steve Brule, For Your Wine',
'description': 'Dr. Brule reports live from Wine Country with a special report on wines. \r\nWatch Tim and Eric Awesome Show Great Job! episode #20, "Embarrassed" on Adult Swim.\r\n\r\n',
},
}
],
'info_dict': {
'id': 'sY3cMUR_TbuE4YmdjzbIcQ',
'title': 'Tim and Eric Awesome Show Great Job! - Dr. Steve Brule, For Your Wine',
'description': 'Dr. Brule reports live from Wine Country with a special report on wines. \r\nWatch Tim and Eric Awesome Show Great Job! episode #20, "Embarrassed" on Adult Swim.\r\n\r\n',
},
}] }]
@staticmethod @staticmethod
@ -80,6 +97,7 @@ class AdultSwimIE(InfoExtractor):
for video in collection.get('videos'): for video in collection.get('videos'):
if video.get('slug') == slug: if video.get('slug') == slug:
return collection, video return collection, video
return None, None
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
@ -90,28 +108,30 @@ class AdultSwimIE(InfoExtractor):
webpage = self._download_webpage(url, episode_path) webpage = self._download_webpage(url, episode_path)
# Extract the value of `bootstrappedData` from the Javascript in the page. # Extract the value of `bootstrappedData` from the Javascript in the page.
bootstrappedDataJS = self._search_regex(r'var bootstrappedData = ({.*});', webpage, episode_path) bootstrapped_data = self._parse_json(self._search_regex(
r'var bootstrappedData = ({.*});', webpage, 'bootstraped data'), episode_path)
try:
bootstrappedData = json.loads(bootstrappedDataJS)
except ValueError as ve:
errmsg = '%s: Failed to parse JSON ' % episode_path
raise ExtractorError(errmsg, cause=ve)
# Downloading videos from a /videos/playlist/ URL needs to be handled differently. # Downloading videos from a /videos/playlist/ URL needs to be handled differently.
# NOTE: We are only downloading one video (the current one) not the playlist # NOTE: We are only downloading one video (the current one) not the playlist
if is_playlist: if is_playlist:
collections = bootstrappedData['playlists']['collections'] collections = bootstrapped_data['playlists']['collections']
collection = self.find_collection_by_linkURL(collections, show_path) collection = self.find_collection_by_linkURL(collections, show_path)
video_info = self.find_video_info(collection, episode_path) video_info = self.find_video_info(collection, episode_path)
show_title = video_info['showTitle'] show_title = video_info['showTitle']
segment_ids = [video_info['videoPlaybackID']] segment_ids = [video_info['videoPlaybackID']]
else: else:
collections = bootstrappedData['show']['collections'] collections = bootstrapped_data['show']['collections']
collection, video_info = self.find_collection_containing_video(collections, episode_path) collection, video_info = self.find_collection_containing_video(collections, episode_path)
show = bootstrappedData['show'] # Video wasn't found in the collections, let's try `slugged_video`.
if video_info is None:
if bootstrapped_data.get('slugged_video', {}).get('slug') == episode_path:
video_info = bootstrapped_data['slugged_video']
else:
raise ExtractorError('Unable to find video info')
show = bootstrapped_data['show']
show_title = show['title'] show_title = show['title']
segment_ids = [clip['videoPlaybackID'] for clip in video_info['clips']] segment_ids = [clip['videoPlaybackID'] for clip in video_info['clips']]

11
youtube_dl/extractor/aftenposten.py
View File

@ -14,10 +14,10 @@ from ..utils import (
class AftenpostenIE(InfoExtractor): class AftenpostenIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?aftenposten\.no/webtv/([^/]+/)*(?P<id>[^/]+)-\d+\.html' _VALID_URL = r'https?://(?:www\.)?aftenposten\.no/webtv/(?:#!/)?video/(?P<id>\d+)'
_TEST = { _TEST = {
'url': 'http://www.aftenposten.no/webtv/serier-og-programmer/sweatshopenglish/TRAILER-SWEATSHOP---I-cant-take-any-more-7800835.html?paging=&section=webtv_serierogprogrammer_sweatshop_sweatshopenglish', 'url': 'http://www.aftenposten.no/webtv/#!/video/21039/trailer-sweatshop-i-can-t-take-any-more',
'md5': 'fd828cd29774a729bf4d4425fe192972', 'md5': 'fd828cd29774a729bf4d4425fe192972',
'info_dict': { 'info_dict': {
'id': '21039', 'id': '21039',
@ -30,12 +30,7 @@ class AftenpostenIE(InfoExtractor):
} }
def _real_extract(self, url): def _real_extract(self, url):
display_id = self._match_id(url) video_id = self._match_id(url)
webpage = self._download_webpage(url, display_id)
video_id = self._html_search_regex(
r'data-xs-id="(\d+)"', webpage, 'video id')
data = self._download_xml( data = self._download_xml(
'http://frontend.xstream.dk/ap/feed/video/?platform=web&id=%s' % video_id, video_id) 'http://frontend.xstream.dk/ap/feed/video/?platform=web&id=%s' % video_id, video_id)

3
youtube_dl/extractor/ard.py
View File

@ -50,6 +50,9 @@ class ARDMediathekIE(InfoExtractor):
if '>Der gewünschte Beitrag ist nicht mehr verfügbar.<' in webpage: if '>Der gewünschte Beitrag ist nicht mehr verfügbar.<' in webpage:
raise ExtractorError('Video %s is no longer available' % video_id, expected=True) raise ExtractorError('Video %s is no longer available' % video_id, expected=True)
if 'Diese Sendung ist für Jugendliche unter 12 Jahren nicht geeignet. Der Clip ist deshalb nur von 20 bis 6 Uhr verfügbar.' in webpage:
raise ExtractorError('This program is only suitable for those aged 12 and older. Video %s is therefore only available between 20 pm and 6 am.' % video_id, expected=True)
if re.search(r'[\?&]rss($|[=&])', url): if re.search(r'[\?&]rss($|[=&])', url):
doc = parse_xml(webpage) doc = parse_xml(webpage)
if doc.tag == 'rss': if doc.tag == 'rss':

1
youtube_dl/extractor/atresplayer.py
View File

@ -19,6 +19,7 @@ from ..utils import (
class AtresPlayerIE(InfoExtractor): class AtresPlayerIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?atresplayer\.com/television/[^/]+/[^/]+/[^/]+/(?P<id>.+?)_\d+\.html' _VALID_URL = r'https?://(?:www\.)?atresplayer\.com/television/[^/]+/[^/]+/[^/]+/(?P<id>.+?)_\d+\.html'
_NETRC_MACHINE = 'atresplayer'
_TESTS = [ _TESTS = [
{ {
'url': 'http://www.atresplayer.com/television/programas/el-club-de-la-comedia/temporada-4/capitulo-10-especial-solidario-nochebuena_2014122100174.html', 'url': 'http://www.atresplayer.com/television/programas/el-club-de-la-comedia/temporada-4/capitulo-10-especial-solidario-nochebuena_2014122100174.html',

103
youtube_dl/extractor/beatportpro.py Normal file
View File

@ -0,0 +1,103 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..compat import compat_str
from ..utils import int_or_none
class BeatportProIE(InfoExtractor):
_VALID_URL = r'https?://pro\.beatport\.com/track/(?P<display_id>[^/]+)/(?P<id>[0-9]+)'
_TESTS = [{
'url': 'https://pro.beatport.com/track/synesthesia-original-mix/5379371',
'md5': 'b3c34d8639a2f6a7f734382358478887',
'info_dict': {
'id': '5379371',
'display_id': 'synesthesia-original-mix',
'ext': 'mp4',
'title': 'Froxic - Synesthesia (Original Mix)',
},
}, {
'url': 'https://pro.beatport.com/track/love-and-war-original-mix/3756896',
'md5': 'e44c3025dfa38c6577fbaeb43da43514',
'info_dict': {
'id': '3756896',
'display_id': 'love-and-war-original-mix',
'ext': 'mp3',
'title': 'Wolfgang Gartner - Love & War (Original Mix)',
},
}, {
'url': 'https://pro.beatport.com/track/birds-original-mix/4991738',
'md5': 'a1fd8e8046de3950fd039304c186c05f',
'info_dict': {
'id': '4991738',
'display_id': 'birds-original-mix',
'ext': 'mp4',
'title': "Tos, Middle Milk, Mumblin' Johnsson - Birds (Original Mix)",
}
}]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
track_id = mobj.group('id')
display_id = mobj.group('display_id')
webpage = self._download_webpage(url, display_id)
playables = self._parse_json(
self._search_regex(
r'window\.Playables\s*=\s*({.+?});', webpage,
'playables info', flags=re.DOTALL),
track_id)
track = next(t for t in playables['tracks'] if t['id'] == int(track_id))
title = ', '.join((a['name'] for a in track['artists'])) + ' - ' + track['name']
if track['mix']:
title += ' (' + track['mix'] + ')'
formats = []
for ext, info in track['preview'].items():
if not info['url']:
continue
fmt = {
'url': info['url'],
'ext': ext,
'format_id': ext,
'vcodec': 'none',
}
if ext == 'mp3':
fmt['preference'] = 0
fmt['acodec'] = 'mp3'
fmt['abr'] = 96
fmt['asr'] = 44100
elif ext == 'mp4':
fmt['preference'] = 1
fmt['acodec'] = 'aac'
fmt['abr'] = 96
fmt['asr'] = 44100
formats.append(fmt)
self._sort_formats(formats)
images = []
for name, info in track['images'].items():
image_url = info.get('url')
if name == 'dynamic' or not image_url:
continue
image = {
'id': name,
'url': image_url,
'height': int_or_none(info.get('height')),
'width': int_or_none(info.get('width')),
}
images.append(image)
return {
'id': compat_str(track.get('id')) or track_id,
'display_id': track.get('slug') or display_id,
'title': title,
'formats': formats,
'thumbnails': images,
}

2
youtube_dl/extractor/breakcom.py
View File

@ -41,7 +41,7 @@ class BreakIE(InfoExtractor):
'tbr': media['bitRate'], 'tbr': media['bitRate'],
'width': media['width'], 'width': media['width'],
'height': media['height'], 'height': media['height'],
} for media in info['media']] } for media in info['media'] if media.get('mediaPurpose') == 'play']
if not formats: if not formats:
formats.append({ formats.append({

3
youtube_dl/extractor/cloudy.py
View File

@ -105,6 +105,7 @@ class CloudyIE(InfoExtractor):
webpage = self._download_webpage(url, video_id) webpage = self._download_webpage(url, video_id)
file_key = self._search_regex( file_key = self._search_regex(
r'filekey\s*=\s*"([^"]+)"', webpage, 'file_key') [r'key\s*:\s*"([^"]+)"', r'filekey\s*=\s*"([^"]+)"'],
webpage, 'file_key')
return self._extract_video(video_host, video_id, file_key) return self._extract_video(video_host, video_id, file_key)

16
youtube_dl/extractor/common.py
View File

@ -771,6 +771,10 @@ class InfoExtractor(object):
formats) formats)
def _is_valid_url(self, url, video_id, item='video'): def _is_valid_url(self, url, video_id, item='video'):
url = self._proto_relative_url(url, scheme='http:')
# For now assume non HTTP(S) URLs always valid
if not (url.startswith('http://') or url.startswith('https://')):
return True
try: try:
self._request_webpage(url, video_id, 'Checking %s URL' % item) self._request_webpage(url, video_id, 'Checking %s URL' % item)
return True return True
@ -862,7 +866,7 @@ class InfoExtractor(object):
m3u8_id=None): m3u8_id=None):
formats = [{ formats = [{
'format_id': '-'.join(filter(None, [m3u8_id, 'm3u8-meta'])), 'format_id': '-'.join(filter(None, [m3u8_id, 'meta'])),
'url': m3u8_url, 'url': m3u8_url,
'ext': ext, 'ext': ext,
'protocol': 'm3u8', 'protocol': 'm3u8',
@ -906,8 +910,13 @@ class InfoExtractor(object):
formats.append({'url': format_url(line)}) formats.append({'url': format_url(line)})
continue continue
tbr = int_or_none(last_info.get('BANDWIDTH'), scale=1000) tbr = int_or_none(last_info.get('BANDWIDTH'), scale=1000)
format_id = []
if m3u8_id:
format_id.append(m3u8_id)
last_media_name = last_media.get('NAME') if last_media else None
format_id.append(last_media_name if last_media_name else '%d' % (tbr if tbr else len(formats)))
f = { f = {
'format_id': '-'.join(filter(None, [m3u8_id, 'm3u8-%d' % (tbr if tbr else len(formats))])), 'format_id': '-'.join(format_id),
'url': format_url(line.strip()), 'url': format_url(line.strip()),
'tbr': tbr, 'tbr': tbr,
'ext': ext, 'ext': ext,
@ -1080,6 +1089,9 @@ class InfoExtractor(object):
def _get_automatic_captions(self, *args, **kwargs): def _get_automatic_captions(self, *args, **kwargs):
raise NotImplementedError("This method must be implemented by subclasses") raise NotImplementedError("This method must be implemented by subclasses")
def _subtitles_timecode(self, seconds):
return '%02d:%02d:%02d.%03d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 1000)
class SearchInfoExtractor(InfoExtractor): class SearchInfoExtractor(InfoExtractor):
""" """

1
youtube_dl/extractor/crunchyroll.py
View File

@ -29,6 +29,7 @@ from ..aes import (
class CrunchyrollIE(InfoExtractor): class CrunchyrollIE(InfoExtractor):
_VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:[^/]*/[^/?&]*?|media/\?id=)(?P<video_id>[0-9]+))(?:[/?&]|$)' _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:[^/]*/[^/?&]*?|media/\?id=)(?P<video_id>[0-9]+))(?:[/?&]|$)'
_NETRC_MACHINE = 'crunchyroll'
_TESTS = [{ _TESTS = [{
'url': 'http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513', 'url': 'http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513',
'info_dict': { 'info_dict': {

10
youtube_dl/extractor/dailymotion.py
View File

@ -46,13 +46,13 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
_TESTS = [ _TESTS = [
{ {
'url': 'http://www.dailymotion.com/video/x33vw9_tutoriel-de-youtubeur-dl-des-video_tech', 'url': 'https://www.dailymotion.com/video/x2iuewm_steam-machine-models-pricing-listed-on-steam-store-ign-news_videogames',
'md5': '392c4b85a60a90dc4792da41ce3144eb', 'md5': '2137c41a8e78554bb09225b8eb322406',
'info_dict': { 'info_dict': {
'id': 'x33vw9', 'id': 'x2iuewm',
'ext': 'mp4', 'ext': 'mp4',
'uploader': 'Amphora Alex and Van .', 'uploader': 'IGN',
'title': 'Tutoriel de Youtubeur"DL DES VIDEO DE YOUTUBE"', 'title': 'Steam Machine Models, Pricing Listed on Steam Store - IGN News',
} }
}, },
# Vevo video # Vevo video

98
youtube_dl/extractor/eagleplatform.py Normal file
View File

@ -0,0 +1,98 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
ExtractorError,
int_or_none,
)
class EaglePlatformIE(InfoExtractor):
_VALID_URL = r'''(?x)
(?:
eagleplatform:(?P<custom_host>[^/]+):|
https?://(?P<host>.+?\.media\.eagleplatform\.com)/index/player\?.*\brecord_id=
)
(?P<id>\d+)
'''
_TESTS = [{
# http://lenta.ru/news/2015/03/06/navalny/
'url': 'http://lentaru.media.eagleplatform.com/index/player?player=new&record_id=227304&player_template_id=5201',
'md5': '0b7994faa2bd5c0f69a3db6db28d078d',
'info_dict': {
'id': '227304',
'ext': 'mp4',
'title': 'Навальный вышел на свободу',
'description': 'md5:d97861ac9ae77377f3f20eaf9d04b4f5',
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 87,
'view_count': int,
'age_limit': 0,
},
}, {
# http://muz-tv.ru/play/7129/
# http://media.clipyou.ru/index/player?record_id=12820&width=730&height=415&autoplay=true
'url': 'eagleplatform:media.clipyou.ru:12820',
'md5': '6c2ebeab03b739597ce8d86339d5a905',
'info_dict': {
'id': '12820',
'ext': 'mp4',
'title': "'O Sole Mio",
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 216,
'view_count': int,
},
}]
def _handle_error(self, response):
status = int_or_none(response.get('status', 200))
if status != 200:
raise ExtractorError(' '.join(response['errors']), expected=True)
def _download_json(self, url_or_request, video_id, note='Downloading JSON metadata'):
response = super(EaglePlatformIE, self)._download_json(url_or_request, video_id, note)
self._handle_error(response)
return response
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
host, video_id = mobj.group('custom_host') or mobj.group('host'), mobj.group('id')
player_data = self._download_json(
'http://%s/api/player_data?id=%s' % (host, video_id), video_id)
media = player_data['data']['playlist']['viewports'][0]['medialist'][0]
title = media['title']
description = media.get('description')
thumbnail = media.get('snapshot')
duration = int_or_none(media.get('duration'))
view_count = int_or_none(media.get('views'))
age_restriction = media.get('age_restriction')
age_limit = None
if age_restriction:
age_limit = 0 if age_restriction == 'allow_all' else 18
m3u8_data = self._download_json(
media['sources']['secure_m3u8']['auto'],
video_id, 'Downloading m3u8 JSON')
formats = self._extract_m3u8_formats(
m3u8_data['data'][0], video_id,
'mp4', entry_protocol='m3u8_native')
self._sort_formats(formats)
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'duration': duration,
'view_count': view_count,
'age_limit': age_limit,
'formats': formats,
}

14
youtube_dl/extractor/eighttracks.py
View File

@ -3,7 +3,6 @@ from __future__ import unicode_literals
import json import json
import random import random
import re
from .common import InfoExtractor from .common import InfoExtractor
from ..compat import ( from ..compat import (
@ -103,20 +102,23 @@ class EightTracksIE(InfoExtractor):
} }
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) playlist_id = self._match_id(url)
playlist_id = mobj.group('id')
webpage = self._download_webpage(url, playlist_id) webpage = self._download_webpage(url, playlist_id)
json_like = self._search_regex( data = self._parse_json(
r"(?s)PAGE.mix = (.*?);\n", webpage, 'trax information') self._search_regex(
data = json.loads(json_like) r"(?s)PAGE\.mix\s*=\s*({.+?});\n", webpage, 'trax information'),
playlist_id)
session = str(random.randint(0, 1000000000)) session = str(random.randint(0, 1000000000))
mix_id = data['id'] mix_id = data['id']
track_count = data['tracks_count'] track_count = data['tracks_count']
duration = data['duration'] duration = data['duration']
avg_song_duration = float(duration) / track_count avg_song_duration = float(duration) / track_count
# duration is sometimes negative, use predefined avg duration
if avg_song_duration <= 0:
avg_song_duration = 300
first_url = 'http://8tracks.com/sets/%s/play?player=sm&mix_id=%s&format=jsonh' % (session, mix_id) first_url = 'http://8tracks.com/sets/%s/play?player=sm&mix_id=%s&format=jsonh' % (session, mix_id)
next_url = first_url next_url = first_url
entries = [] entries = []

30
youtube_dl/extractor/extremetube.py
View File

@ -4,11 +4,11 @@ import re
from .common import InfoExtractor from .common import InfoExtractor
from ..compat import ( from ..compat import (
compat_urllib_parse_urlparse, compat_parse_qs,
compat_urllib_request, compat_urllib_request,
compat_urllib_parse,
) )
from ..utils import ( from ..utils import (
qualities,
str_to_int, str_to_int,
) )
@ -17,7 +17,7 @@ class ExtremeTubeIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?(?P<url>extremetube\.com/.*?video/.+?(?P<id>[0-9]+))(?:[/?&]|$)' _VALID_URL = r'https?://(?:www\.)?(?P<url>extremetube\.com/.*?video/.+?(?P<id>[0-9]+))(?:[/?&]|$)'
_TESTS = [{ _TESTS = [{
'url': 'http://www.extremetube.com/video/music-video-14-british-euro-brit-european-cumshots-swallow-652431', 'url': 'http://www.extremetube.com/video/music-video-14-british-euro-brit-european-cumshots-swallow-652431',
'md5': '1fb9228f5e3332ec8c057d6ac36f33e0', 'md5': '344d0c6d50e2f16b06e49ca011d8ac69',
'info_dict': { 'info_dict': {
'id': '652431', 'id': '652431',
'ext': 'mp4', 'ext': 'mp4',
@ -49,19 +49,27 @@ class ExtremeTubeIE(InfoExtractor):
r'Views:\s*</strong>\s*<span>([\d,\.]+)</span>', r'Views:\s*</strong>\s*<span>([\d,\.]+)</span>',
webpage, 'view count', fatal=False)) webpage, 'view count', fatal=False))
video_url = compat_urllib_parse.unquote(self._html_search_regex( flash_vars = compat_parse_qs(self._search_regex(
r'video_url=(.+?)&amp;', webpage, 'video_url')) r'<param[^>]+?name="flashvars"[^>]+?value="([^"]+)"', webpage, 'flash vars'))
path = compat_urllib_parse_urlparse(video_url).path
format = path.split('/')[5].split('_')[:2] formats = []
format = "-".join(format) quality = qualities(['180p', '240p', '360p', '480p', '720p', '1080p'])
for k, vals in flash_vars.items():
m = re.match(r'quality_(?P<quality>[0-9]+p)$', k)
if m is not None:
formats.append({
'format_id': m.group('quality'),
'quality': quality(m.group('quality')),
'url': vals[0],
})
self._sort_formats(formats)
return { return {
'id': video_id, 'id': video_id,
'title': video_title, 'title': video_title,
'formats': formats,
'uploader': uploader, 'uploader': uploader,
'view_count': view_count, 'view_count': view_count,
'url': video_url,
'format': format,
'format_id': format,
'age_limit': 18, 'age_limit': 18,
} }

41
youtube_dl/extractor/footyroom.py Normal file
View File

@ -0,0 +1,41 @@
# coding: utf-8
from __future__ import unicode_literals
from .common import InfoExtractor
class FootyRoomIE(InfoExtractor):
_VALID_URL = r'http://footyroom\.com/(?P<id>[^/]+)'
_TEST = {
'url': 'http://footyroom.com/schalke-04-0-2-real-madrid-2015-02/',
'info_dict': {
'id': 'schalke-04-0-2-real-madrid-2015-02',
'title': 'Schalke 04 0 – 2 Real Madrid',
},
'playlist_count': 3,
}
def _real_extract(self, url):
playlist_id = self._match_id(url)
webpage = self._download_webpage(url, playlist_id)
playlist = self._parse_json(
self._search_regex(
r'VideoSelector\.load\((\[.+?\])\);', webpage, 'video selector'),
playlist_id)
playlist_title = self._og_search_title(webpage)
entries = []
for video in playlist:
payload = video.get('payload')
if not payload:
continue
playwire_url = self._search_regex(
r'data-config="([^"]+)"', payload,
'playwire url', default=None)
if playwire_url:
entries.append(self.url_result(playwire_url, 'Playwire'))
return self.playlist_result(entries, playlist_id, playlist_title)

9
youtube_dl/extractor/funnyordie.py
View File

@ -50,7 +50,6 @@ class FunnyOrDieIE(InfoExtractor):
bitrates.sort() bitrates.sort()
formats = [] formats = []
for bitrate in bitrates: for bitrate in bitrates:
for link in links: for link in links:
formats.append({ formats.append({
@ -59,6 +58,13 @@ class FunnyOrDieIE(InfoExtractor):
'vbr': bitrate, 'vbr': bitrate,
}) })
subtitles = {}
for src, src_lang in re.findall(r'<track kind="captions" src="([^"]+)" srclang="([^"]+)"', webpage):
subtitles[src_lang] = [{
'ext': src.split('/')[-1],
'url': 'http://www.funnyordie.com%s' % src,
}]
post_json = self._search_regex( post_json = self._search_regex(
r'fb_post\s*=\s*(\{.*?\});', webpage, 'post details') r'fb_post\s*=\s*(\{.*?\});', webpage, 'post details')
post = json.loads(post_json) post = json.loads(post_json)
@ -69,4 +75,5 @@ class FunnyOrDieIE(InfoExtractor):
'description': post.get('description'), 'description': post.get('description'),
'thumbnail': post.get('picture'), 'thumbnail': post.get('picture'),
'formats': formats, 'formats': formats,
'subtitles': subtitles,
} }

38
youtube_dl/extractor/gazeta.py Normal file
View File

@ -0,0 +1,38 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
class GazetaIE(InfoExtractor):
_VALID_URL = r'(?P<url>https?://(?:www\.)?gazeta\.ru/(?:[^/]+/)?video/(?:(?:main|\d{4}/\d{2}/\d{2})/)?(?P<id>[A-Za-z0-9-_.]+)\.s?html)'
_TESTS = [{
'url': 'http://www.gazeta.ru/video/main/zadaite_vopros_vladislavu_yurevichu.shtml',
'md5': 'd49c9bdc6e5a7888f27475dc215ee789',
'info_dict': {
'id': '205566',
'ext': 'mp4',
'title': '«70–80 процентов гражданских в Донецке на грани голода»',
'description': 'md5:38617526050bd17b234728e7f9620a71',
'thumbnail': 're:^https?://.*\.jpg',
},
}, {
'url': 'http://www.gazeta.ru/lifestyle/video/2015/03/08/master-klass_krasivoi_byt._delaem_vesennii_makiyazh.shtml',
'only_matching': True,
}]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
display_id = mobj.group('id')
embed_url = '%s?p=embed' % mobj.group('url')
embed_page = self._download_webpage(
embed_url, display_id, 'Downloading embed page')
video_id = self._search_regex(
r'<div[^>]*?class="eagleplayer"[^>]*?data-id="([^"]+)"', embed_page, 'video id')
return self.url_result(
'eagleplatform:gazeta.media.eagleplatform.com:%s' % video_id, 'EaglePlatform')

1
youtube_dl/extractor/gdcvault.py
View File

@ -12,6 +12,7 @@ from ..utils import remove_end
class GDCVaultIE(InfoExtractor): class GDCVaultIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?gdcvault\.com/play/(?P<id>\d+)/(?P<name>(\w|-)+)' _VALID_URL = r'https?://(?:www\.)?gdcvault\.com/play/(?P<id>\d+)/(?P<name>(\w|-)+)'
_NETRC_MACHINE = 'gdcvault'
_TESTS = [ _TESTS = [
{ {
'url': 'http://www.gdcvault.com/play/1019721/Doki-Doki-Universe-Sweet-Simple', 'url': 'http://www.gdcvault.com/play/1019721/Doki-Doki-Universe-Sweet-Simple',

91
youtube_dl/extractor/generic.py
View File

@ -26,6 +26,7 @@ from ..utils import (
unsmuggle_url, unsmuggle_url,
UnsupportedError, UnsupportedError,
url_basename, url_basename,
xpath_text,
) )
from .brightcove import BrightcoveIE from .brightcove import BrightcoveIE
from .ooyala import OoyalaIE from .ooyala import OoyalaIE
@ -569,6 +570,55 @@ class GenericIE(InfoExtractor):
'title': 'John Carlson Postgame 2/25/15', 'title': 'John Carlson Postgame 2/25/15',
}, },
}, },
# Eagle.Platform embed (generic URL)
{
'url': 'http://lenta.ru/news/2015/03/06/navalny/',
'info_dict': {
'id': '227304',
'ext': 'mp4',
'title': 'Навальный вышел на свободу',
'description': 'md5:d97861ac9ae77377f3f20eaf9d04b4f5',
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 87,
'view_count': int,
'age_limit': 0,
},
},
# ClipYou (Eagle.Platform) embed (custom URL)
{
'url': 'http://muz-tv.ru/play/7129/',
'info_dict': {
'id': '12820',
'ext': 'mp4',
'title': "'O Sole Mio",
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 216,
'view_count': int,
},
},
# Pladform embed
{
'url': 'http://muz-tv.ru/kinozal/view/7400/',
'info_dict': {
'id': '100183293',
'ext': 'mp4',
'title': 'Тайны перевала Дятлова • Тайна перевала Дятлова 1 серия 2 часть',
'description': 'Документальный сериал-расследование одной из самых жутких тайн ХХ века',
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 694,
'age_limit': 0,
},
},
# RSS feed with enclosure
{
'url': 'http://podcastfeeds.nbcnews.com/audio/podcast/MSNBC-MADDOW-NETCAST-M4V.xml',
'info_dict': {
'id': 'pdv_maddow_netcast_m4v-02-27-2015-201624',
'ext': 'm4v',
'upload_date': '20150228',
'title': 'pdv_maddow_netcast_m4v-02-27-2015-201624',
}
}
] ]
def report_following_redirect(self, new_url): def report_following_redirect(self, new_url):
@ -580,11 +630,24 @@ class GenericIE(InfoExtractor):
playlist_desc_el = doc.find('./channel/description') playlist_desc_el = doc.find('./channel/description')
playlist_desc = None if playlist_desc_el is None else playlist_desc_el.text playlist_desc = None if playlist_desc_el is None else playlist_desc_el.text
entries = [{ entries = []
'_type': 'url', for it in doc.findall('./channel/item'):
'url': e.find('link').text, next_url = xpath_text(it, 'link', fatal=False)
'title': e.find('title').text, if not next_url:
} for e in doc.findall('./channel/item')] enclosure_nodes = it.findall('./enclosure')
for e in enclosure_nodes:
next_url = e.attrib.get('url')
if next_url:
break
if not next_url:
continue
entries.append({
'_type': 'url',
'url': next_url,
'title': it.find('title').text,
})
return { return {
'_type': 'playlist', '_type': 'playlist',
@ -1131,6 +1194,24 @@ class GenericIE(InfoExtractor):
if mobj is not None: if mobj is not None:
return self.url_result('kaltura:%(partner_id)s:%(id)s' % mobj.groupdict(), 'Kaltura') return self.url_result('kaltura:%(partner_id)s:%(id)s' % mobj.groupdict(), 'Kaltura')
# Look for Eagle.Platform embeds
mobj = re.search(
r'<iframe[^>]+src="(?P<url>https?://.+?\.media\.eagleplatform\.com/index/player\?.+?)"', webpage)
if mobj is not None:
return self.url_result(mobj.group('url'), 'EaglePlatform')
# Look for ClipYou (uses Eagle.Platform) embeds
mobj = re.search(
r'<iframe[^>]+src="https?://(?P<host>media\.clipyou\.ru)/index/player\?.*\brecord_id=(?P<id>\d+).*"', webpage)
if mobj is not None:
return self.url_result('eagleplatform:%(host)s:%(id)s' % mobj.groupdict(), 'EaglePlatform')
# Look for Pladform embeds
mobj = re.search(
r'<iframe[^>]+src="(?P<url>https?://out\.pladform\.ru/player\?.+?)"', webpage)
if mobj is not None:
return self.url_result(mobj.group('url'), 'Pladform')
def check_video(vurl): def check_video(vurl):
if YoutubeIE.suitable(vurl): if YoutubeIE.suitable(vurl):
return True return True

2
youtube_dl/extractor/globo.py
View File

@ -20,7 +20,7 @@ class GloboIE(InfoExtractor):
_VALID_URL = 'https?://.+?\.globo\.com/(?P<id>.+)' _VALID_URL = 'https?://.+?\.globo\.com/(?P<id>.+)'
_API_URL_TEMPLATE = 'http://api.globovideos.com/videos/%s/playlist' _API_URL_TEMPLATE = 'http://api.globovideos.com/videos/%s/playlist'
_SECURITY_URL_TEMPLATE = 'http://security.video.globo.com/videos/%s/hash?player=flash&version=2.9.9.50&resource_id=%s' _SECURITY_URL_TEMPLATE = 'http://security.video.globo.com/videos/%s/hash?player=flash&version=17.0.0.132&resource_id=%s'
_VIDEOID_REGEXES = [ _VIDEOID_REGEXES = [
r'\bdata-video-id="(\d+)"', r'\bdata-video-id="(\d+)"',

36
youtube_dl/extractor/jeuxvideo.py
View File

@ -2,7 +2,6 @@
from __future__ import unicode_literals from __future__ import unicode_literals
import json
import re import re
from .common import InfoExtractor from .common import InfoExtractor
@ -15,10 +14,10 @@ class JeuxVideoIE(InfoExtractor):
'url': 'http://www.jeuxvideo.com/reportages-videos-jeux/0004/00046170/tearaway-playstation-vita-gc-2013-tearaway-nous-presente-ses-papiers-d-identite-00115182.htm', 'url': 'http://www.jeuxvideo.com/reportages-videos-jeux/0004/00046170/tearaway-playstation-vita-gc-2013-tearaway-nous-presente-ses-papiers-d-identite-00115182.htm',
'md5': '046e491afb32a8aaac1f44dd4ddd54ee', 'md5': '046e491afb32a8aaac1f44dd4ddd54ee',
'info_dict': { 'info_dict': {
'id': '5182', 'id': '114765',
'ext': 'mp4', 'ext': 'mp4',
'title': 'GC 2013 : Tearaway nous présente ses papiers d\'identité', 'title': 'Tearaway : GC 2013 : Tearaway nous présente ses papiers d\'identité',
'description': 'Lorsque les développeurs de LittleBigPlanet proposent un nouveau titre, on ne peut que s\'attendre à un résultat original et fort attrayant.\n', 'description': 'Lorsque les développeurs de LittleBigPlanet proposent un nouveau titre, on ne peut que s\'attendre à un résultat original et fort attrayant.',
}, },
} }
@ -26,26 +25,29 @@ class JeuxVideoIE(InfoExtractor):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
title = mobj.group(1) title = mobj.group(1)
webpage = self._download_webpage(url, title) webpage = self._download_webpage(url, title)
xml_link = self._html_search_regex( title = self._html_search_meta('name', webpage)
r'<param name="flashvars" value="config=(.*?)" />', config_url = self._html_search_regex(
r'data-src="(/contenu/medias/video.php.*?)"',
webpage, 'config URL') webpage, 'config URL')
config_url = 'http://www.jeuxvideo.com' + config_url
video_id = self._search_regex( video_id = self._search_regex(
r'http://www\.jeuxvideo\.com/config/\w+/\d+/(.*?)/\d+_player\.xml', r'id=(\d+)',
xml_link, 'video ID') config_url, 'video ID')
config = self._download_xml( config = self._download_json(
xml_link, title, 'Downloading XML config') config_url, title, 'Downloading JSON config')
info_json = config.find('format.json').text
info = json.loads(info_json)['versions'][0]
video_url = 'http://video720.jeuxvideo.com/' + info['file'] formats = [{
'url': source['file'],
'format_id': source['label'],
'resolution': source['label'],
} for source in reversed(config['sources'])]
return { return {
'id': video_id, 'id': video_id,
'title': config.find('titre_video').text, 'title': title,
'ext': 'mp4', 'formats': formats,
'url': video_url,
'description': self._og_search_description(webpage), 'description': self._og_search_description(webpage),
'thumbnail': config.find('image').text, 'thumbnail': config.get('image'),
} }

96
youtube_dl/extractor/kanalplay.py Normal file
View File

@ -0,0 +1,96 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
ExtractorError,
float_or_none,
)
class KanalPlayIE(InfoExtractor):
IE_DESC = 'Kanal 5/9/11 Play'
_VALID_URL = r'https?://(?:www\.)?kanal(?P<channel_id>5|9|11)play\.se/(?:#!/)?(?:play/)?program/\d+/video/(?P<id>\d+)'
_TESTS = [{
'url': 'http://www.kanal5play.se/#!/play/program/3060212363/video/3270012277',
'info_dict': {
'id': '3270012277',
'ext': 'flv',
'title': 'Saknar både dusch och avlopp',
'description': 'md5:6023a95832a06059832ae93bc3c7efb7',
'duration': 2636.36,
},
'params': {
# rtmp download
'skip_download': True,
}
}, {
'url': 'http://www.kanal9play.se/#!/play/program/335032/video/246042',
'only_matching': True,
}, {
'url': 'http://www.kanal11play.se/#!/play/program/232835958/video/367135199',
'only_matching': True,
}]
def _fix_subtitles(self, subs):
return '\r\n\r\n'.join(
'%s\r\n%s --> %s\r\n%s'
% (
num,
self._subtitles_timecode(item['startMillis'] / 1000.0),
self._subtitles_timecode(item['endMillis'] / 1000.0),
item['text'],
) for num, item in enumerate(subs, 1))
def _get_subtitles(self, channel_id, video_id):
subs = self._download_json(
'http://www.kanal%splay.se/api/subtitles/%s' % (channel_id, video_id),
video_id, 'Downloading subtitles JSON', fatal=False)
return {'se': [{'ext': 'srt', 'data': self._fix_subtitles(subs)}]} if subs else {}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
channel_id = mobj.group('channel_id')
video = self._download_json(
'http://www.kanal%splay.se/api/getVideo?format=FLASH&videoId=%s' % (channel_id, video_id),
video_id)
reasons_for_no_streams = video.get('reasonsForNoStreams')
if reasons_for_no_streams:
raise ExtractorError(
'%s returned error: %s' % (self.IE_NAME, '\n'.join(reasons_for_no_streams)),
expected=True)
title = video['title']
description = video.get('description')
duration = float_or_none(video.get('length'), 1000)
thumbnail = video.get('posterUrl')
stream_base_url = video['streamBaseUrl']
formats = [{
'url': stream_base_url,
'play_path': stream['source'],
'ext': 'flv',
'tbr': float_or_none(stream.get('bitrate'), 1000),
'rtmp_real_time': True,
} for stream in video['streams']]
self._sort_formats(formats)
subtitles = {}
if video.get('hasSubtitle'):
subtitles = self.extract_subtitles(channel_id, video_id)
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'duration': duration,
'formats': formats,
'subtitles': subtitles,
}

34
youtube_dl/extractor/letv.py
View File

@ -7,8 +7,9 @@ import time
from .common import InfoExtractor from .common import InfoExtractor
from ..compat import ( from ..compat import (
compat_urlparse,
compat_urllib_parse, compat_urllib_parse,
compat_urllib_request,
compat_urlparse,
) )
from ..utils import ( from ..utils import (
determine_ext, determine_ext,
@ -39,12 +40,20 @@ class LetvIE(InfoExtractor):
'title': '美人天下01', 'title': '美人天下01',
'description': 'md5:f88573d9d7225ada1359eaf0dbf8bcda', 'description': 'md5:f88573d9d7225ada1359eaf0dbf8bcda',
}, },
'expected_warnings': [ }, {
'publish time' 'note': 'This video is available only in Mainland China, thus a proxy is needed',
] 'url': 'http://www.letv.com/ptv/vplay/1118082.html',
'md5': 'f80936fbe20fb2f58648e81386ff7927',
'info_dict': {
'id': '1118082',
'ext': 'mp4',
'title': '与龙共舞 完整版',
'description': 'md5:7506a5eeb1722bb9d4068f85024e3986',
},
'params': {
'cn_verification_proxy': 'http://proxy.uku.im:8888'
},
}] }]
# http://www.letv.com/ptv/vplay/1118082.html
# This video is available only in Mainland China
@staticmethod @staticmethod
def urshift(val, n): def urshift(val, n):
@ -76,8 +85,14 @@ class LetvIE(InfoExtractor):
'tkey': self.calc_time_key(int(time.time())), 'tkey': self.calc_time_key(int(time.time())),
'domain': 'www.letv.com' 'domain': 'www.letv.com'
} }
play_json_req = compat_urllib_request.Request(
'http://api.letv.com/mms/out/video/playJson?' + compat_urllib_parse.urlencode(params)
)
play_json_req.add_header(
'Ytdl-request-proxy',
self._downloader.params.get('cn_verification_proxy'))
play_json = self._download_json( play_json = self._download_json(
'http://api.letv.com/mms/out/video/playJson?' + compat_urllib_parse.urlencode(params), play_json_req,
media_id, 'playJson data') media_id, 'playJson data')
# Check for errors # Check for errors
@ -114,7 +129,8 @@ class LetvIE(InfoExtractor):
url_info_dict = { url_info_dict = {
'url': media_url, 'url': media_url,
'ext': determine_ext(dispatch[format_id][1]) 'ext': determine_ext(dispatch[format_id][1]),
'format_id': format_id,
} }
if format_id[-1:] == 'p': if format_id[-1:] == 'p':
@ -123,7 +139,7 @@ class LetvIE(InfoExtractor):
urls.append(url_info_dict) urls.append(url_info_dict)
publish_time = parse_iso8601(self._html_search_regex( publish_time = parse_iso8601(self._html_search_regex(
r'发布时间&nbsp;([^<>]+) ', page, 'publish time', fatal=False), r'发布时间&nbsp;([^<>]+) ', page, 'publish time', default=None),
delimiter=' ', timezone=datetime.timedelta(hours=8)) delimiter=' ', timezone=datetime.timedelta(hours=8))
description = self._html_search_meta('description', page, fatal=False) description = self._html_search_meta('description', page, fatal=False)

41
youtube_dl/extractor/livestream.py
View File

@ -2,6 +2,7 @@ from __future__ import unicode_literals
import re import re
import json import json
import itertools
from .common import InfoExtractor from .common import InfoExtractor
from ..compat import ( from ..compat import (
@ -40,6 +41,13 @@ class LivestreamIE(InfoExtractor):
'id': '2245590', 'id': '2245590',
}, },
'playlist_mincount': 4, 'playlist_mincount': 4,
}, {
'url': 'http://new.livestream.com/chess24/tatasteelchess',
'info_dict': {
'title': 'Tata Steel Chess',
'id': '3705884',
},
'playlist_mincount': 60,
}, { }, {
'url': 'https://new.livestream.com/accounts/362/events/3557232/videos/67864563/player?autoPlay=false&height=360&mute=false&width=640', 'url': 'https://new.livestream.com/accounts/362/events/3557232/videos/67864563/player?autoPlay=false&height=360&mute=false&width=640',
'only_matching': True, 'only_matching': True,
@ -117,6 +125,30 @@ class LivestreamIE(InfoExtractor):
'view_count': video_data.get('views'), 'view_count': video_data.get('views'),
} }
def _extract_event(self, info):
event_id = compat_str(info['id'])
account = compat_str(info['owner_account_id'])
root_url = (
'https://new.livestream.com/api/accounts/{account}/events/{event}/'
'feed.json'.format(account=account, event=event_id))
def _extract_videos():
last_video = None
for i in itertools.count(1):
if last_video is None:
info_url = root_url
else:
info_url = '{root}?&id={id}&newer=-1&type=video'.format(
root=root_url, id=last_video)
videos_info = self._download_json(info_url, event_id, 'Downloading page {0}'.format(i))['data']
videos_info = [v['data'] for v in videos_info if v['type'] == 'video']
if not videos_info:
break
for v in videos_info:
yield self._extract_video_info(v)
last_video = videos_info[-1]['id']
return self.playlist_result(_extract_videos(), event_id, info['full_name'])
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id') video_id = mobj.group('id')
@ -144,14 +176,13 @@ class LivestreamIE(InfoExtractor):
result = result and compat_str(vdata['data']['id']) == vid result = result and compat_str(vdata['data']['id']) == vid
return result return result
videos = [self._extract_video_info(video_data['data'])
for video_data in info['feed']['data']
if is_relevant(video_data, video_id)]
if video_id is None: if video_id is None:
# This is an event page: # This is an event page:
return self.playlist_result( return self._extract_event(info)
videos, '%s' % info['id'], info['full_name'])
else: else:
videos = [self._extract_video_info(video_data['data'])
for video_data in info['feed']['data']
if is_relevant(video_data, video_id)]
if not videos: if not videos:
raise ExtractorError('Cannot find video %s' % video_id) raise ExtractorError('Cannot find video %s' % video_id)
return videos[0] return videos[0]

1
youtube_dl/extractor/lrt.py
View File

@ -52,6 +52,7 @@ class LRTIE(InfoExtractor):
'url': data['streamer'], 'url': data['streamer'],
'play_path': 'mp4:%s' % data['file'], 'play_path': 'mp4:%s' % data['file'],
'preference': -1, 'preference': -1,
'rtmp_real_time': True,
}) })
else: else:
formats.extend( formats.extend(

159
youtube_dl/extractor/lynda.py
View File

@ -15,17 +15,72 @@ from ..utils import (
) )
class LyndaIE(InfoExtractor): class LyndaBaseIE(InfoExtractor):
IE_NAME = 'lynda'
IE_DESC = 'lynda.com videos'
_VALID_URL = r'https?://www\.lynda\.com/(?:[^/]+/[^/]+/\d+|player/embed)/(\d+)'
_LOGIN_URL = 'https://www.lynda.com/login/login.aspx' _LOGIN_URL = 'https://www.lynda.com/login/login.aspx'
_SUCCESSFUL_LOGIN_REGEX = r'isLoggedIn: true'
_ACCOUNT_CREDENTIALS_HINT = 'Use --username and --password options to provide lynda.com account credentials.'
_NETRC_MACHINE = 'lynda' _NETRC_MACHINE = 'lynda'
_SUCCESSFUL_LOGIN_REGEX = r'isLoggedIn: true' def _real_initialize(self):
_TIMECODE_REGEX = r'\[(?P<timecode>\d+:\d+:\d+[\.,]\d+)\]' self._login()
ACCOUNT_CREDENTIALS_HINT = 'Use --username and --password options to provide lynda.com account credentials.' def _login(self):
(username, password) = self._get_login_info()
if username is None:
return
login_form = {
'username': username,
'password': password,
'remember': 'false',
'stayPut': 'false'
}
request = compat_urllib_request.Request(
self._LOGIN_URL, compat_urllib_parse.urlencode(login_form))
login_page = self._download_webpage(
request, None, 'Logging in as %s' % username)
# Not (yet) logged in
m = re.search(r'loginResultJson = \'(?P<json>[^\']+)\';', login_page)
if m is not None:
response = m.group('json')
response_json = json.loads(response)
state = response_json['state']
if state == 'notlogged':
raise ExtractorError(
'Unable to login, incorrect username and/or password',
expected=True)
# This is when we get popup:
# > You're already logged in to lynda.com on two devices.
# > If you log in here, we'll log you out of another device.
# So, we need to confirm this.
if state == 'conflicted':
confirm_form = {
'username': '',
'password': '',
'resolve': 'true',
'remember': 'false',
'stayPut': 'false',
}
request = compat_urllib_request.Request(
self._LOGIN_URL, compat_urllib_parse.urlencode(confirm_form))
login_page = self._download_webpage(
request, None,
'Confirming log in and log out from another device')
if re.search(self._SUCCESSFUL_LOGIN_REGEX, login_page) is None:
raise ExtractorError('Unable to log in')
class LyndaIE(LyndaBaseIE):
IE_NAME = 'lynda'
IE_DESC = 'lynda.com videos'
_VALID_URL = r'https?://www\.lynda\.com/(?:[^/]+/[^/]+/\d+|player/embed)/(?P<id>\d+)'
_NETRC_MACHINE = 'lynda'
_TIMECODE_REGEX = r'\[(?P<timecode>\d+:\d+:\d+[\.,]\d+)\]'
_TESTS = [{ _TESTS = [{
'url': 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html', 'url': 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html',
@ -41,23 +96,22 @@ class LyndaIE(InfoExtractor):
'only_matching': True, 'only_matching': True,
}] }]
def _real_initialize(self):
self._login()
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) video_id = self._match_id(url)
video_id = mobj.group(1)
page = self._download_webpage('http://www.lynda.com/ajax/player?videoId=%s&type=video' % video_id, video_id, page = self._download_webpage(
'Downloading video JSON') 'http://www.lynda.com/ajax/player?videoId=%s&type=video' % video_id,
video_id, 'Downloading video JSON')
video_json = json.loads(page) video_json = json.loads(page)
if 'Status' in video_json: if 'Status' in video_json:
raise ExtractorError('lynda returned error: %s' % video_json['Message'], expected=True) raise ExtractorError(
'lynda returned error: %s' % video_json['Message'], expected=True)
if video_json['HasAccess'] is False: if video_json['HasAccess'] is False:
raise ExtractorError( raise ExtractorError(
'Video %s is only available for members. ' % video_id + self.ACCOUNT_CREDENTIALS_HINT, expected=True) 'Video %s is only available for members. '
% video_id + self._ACCOUNT_CREDENTIALS_HINT, expected=True)
video_id = compat_str(video_json['ID']) video_id = compat_str(video_json['ID'])
duration = video_json['DurationInSeconds'] duration = video_json['DurationInSeconds']
@ -100,50 +154,9 @@ class LyndaIE(InfoExtractor):
'formats': formats 'formats': formats
} }
def _login(self):
(username, password) = self._get_login_info()
if username is None:
return
login_form = {
'username': username,
'password': password,
'remember': 'false',
'stayPut': 'false'
}
request = compat_urllib_request.Request(self._LOGIN_URL, compat_urllib_parse.urlencode(login_form))
login_page = self._download_webpage(request, None, 'Logging in as %s' % username)
# Not (yet) logged in
m = re.search(r'loginResultJson = \'(?P<json>[^\']+)\';', login_page)
if m is not None:
response = m.group('json')
response_json = json.loads(response)
state = response_json['state']
if state == 'notlogged':
raise ExtractorError('Unable to login, incorrect username and/or password', expected=True)
# This is when we get popup:
# > You're already logged in to lynda.com on two devices.
# > If you log in here, we'll log you out of another device.
# So, we need to confirm this.
if state == 'conflicted':
confirm_form = {
'username': '',
'password': '',
'resolve': 'true',
'remember': 'false',
'stayPut': 'false',
}
request = compat_urllib_request.Request(self._LOGIN_URL, compat_urllib_parse.urlencode(confirm_form))
login_page = self._download_webpage(request, None, 'Confirming log in and log out from another device')
if re.search(self._SUCCESSFUL_LOGIN_REGEX, login_page) is None:
raise ExtractorError('Unable to log in')
def _fix_subtitles(self, subs): def _fix_subtitles(self, subs):
srt = '' srt = ''
seq_counter = 0
for pos in range(0, len(subs) - 1): for pos in range(0, len(subs) - 1):
seq_current = subs[pos] seq_current = subs[pos]
m_current = re.match(self._TIMECODE_REGEX, seq_current['Timecode']) m_current = re.match(self._TIMECODE_REGEX, seq_current['Timecode'])
@ -155,8 +168,10 @@ class LyndaIE(InfoExtractor):
continue continue
appear_time = m_current.group('timecode') appear_time = m_current.group('timecode')
disappear_time = m_next.group('timecode') disappear_time = m_next.group('timecode')
text = seq_current['Caption'].lstrip() text = seq_current['Caption'].strip()
srt += '%s\r\n%s --> %s\r\n%s' % (str(pos), appear_time, disappear_time, text) if text:
seq_counter += 1
srt += '%s\r\n%s --> %s\r\n%s\r\n\r\n' % (seq_counter, appear_time, disappear_time, text)
if srt: if srt:
return srt return srt
@ -169,7 +184,7 @@ class LyndaIE(InfoExtractor):
return {} return {}
class LyndaCourseIE(InfoExtractor): class LyndaCourseIE(LyndaBaseIE):
IE_NAME = 'lynda:course' IE_NAME = 'lynda:course'
IE_DESC = 'lynda.com online courses' IE_DESC = 'lynda.com online courses'
@ -182,35 +197,37 @@ class LyndaCourseIE(InfoExtractor):
course_path = mobj.group('coursepath') course_path = mobj.group('coursepath')
course_id = mobj.group('courseid') course_id = mobj.group('courseid')
page = self._download_webpage('http://www.lynda.com/ajax/player?courseId=%s&type=course' % course_id, page = self._download_webpage(
course_id, 'Downloading course JSON') 'http://www.lynda.com/ajax/player?courseId=%s&type=course' % course_id,
course_id, 'Downloading course JSON')
course_json = json.loads(page) course_json = json.loads(page)
if 'Status' in course_json and course_json['Status'] == 'NotFound': if 'Status' in course_json and course_json['Status'] == 'NotFound':
raise ExtractorError('Course %s does not exist' % course_id, expected=True) raise ExtractorError(
'Course %s does not exist' % course_id, expected=True)
unaccessible_videos = 0 unaccessible_videos = 0
videos = [] videos = []
(username, _) = self._get_login_info()
# Might want to extract videos right here from video['Formats'] as it seems 'Formats' is not provided # Might want to extract videos right here from video['Formats'] as it seems 'Formats' is not provided
# by single video API anymore # by single video API anymore
for chapter in course_json['Chapters']: for chapter in course_json['Chapters']:
for video in chapter['Videos']: for video in chapter['Videos']:
if username is None and video['HasAccess'] is False: if video['HasAccess'] is False:
unaccessible_videos += 1 unaccessible_videos += 1
continue continue
videos.append(video['ID']) videos.append(video['ID'])
if unaccessible_videos > 0: if unaccessible_videos > 0:
self._downloader.report_warning('%s videos are only available for members and will not be downloaded. ' self._downloader.report_warning(
% unaccessible_videos + LyndaIE.ACCOUNT_CREDENTIALS_HINT) '%s videos are only available for members (or paid members) and will not be downloaded. '
% unaccessible_videos + self._ACCOUNT_CREDENTIALS_HINT)
entries = [ entries = [
self.url_result('http://www.lynda.com/%s/%s-4.html' % self.url_result(
(course_path, video_id), 'http://www.lynda.com/%s/%s-4.html' % (course_path, video_id),
'Lynda') 'Lynda')
for video_id in videos] for video_id in videos]
course_title = course_json['Title'] course_title = course_json['Title']

5
youtube_dl/extractor/niconico.py
View File

@ -41,7 +41,7 @@ class NiconicoIE(InfoExtractor):
}, },
} }
_VALID_URL = r'https?://(?:www\.|secure\.)?nicovideo\.jp/watch/((?:[a-z]{2})?[0-9]+)' _VALID_URL = r'https?://(?:www\.|secure\.)?nicovideo\.jp/watch/(?P<id>(?:[a-z]{2})?[0-9]+)'
_NETRC_MACHINE = 'niconico' _NETRC_MACHINE = 'niconico'
# Determine whether the downloader used authentication to download video # Determine whether the downloader used authentication to download video
_AUTHENTICATED = False _AUTHENTICATED = False
@ -76,8 +76,7 @@ class NiconicoIE(InfoExtractor):
return True return True
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) video_id = self._match_id(url)
video_id = mobj.group(1)
# Get video webpage. We are not actually interested in it, but need # Get video webpage. We are not actually interested in it, but need
# the cookies in order to be able to download the info webpage # the cookies in order to be able to download the info webpage

4
youtube_dl/extractor/npo.py
View File

@ -219,7 +219,8 @@ class NPOLiveIE(NPOBaseIE):
if streams: if streams:
for stream in streams: for stream in streams:
stream_type = stream.get('type').lower() stream_type = stream.get('type').lower()
if stream_type == 'ss': # smooth streaming is not supported
if stream_type in ['ss', 'ms']:
continue continue
stream_info = self._download_json( stream_info = self._download_json(
'http://ida.omroep.nl/aapi/?stream=%s&token=%s&type=jsonp' 'http://ida.omroep.nl/aapi/?stream=%s&token=%s&type=jsonp'
@ -242,6 +243,7 @@ class NPOLiveIE(NPOBaseIE):
else: else:
formats.append({ formats.append({
'url': stream_url, 'url': stream_url,
'preference': -10,
}) })
self._sort_formats(formats) self._sort_formats(formats)

7
youtube_dl/extractor/nrk.py
View File

@ -149,9 +149,6 @@ class NRKTVIE(InfoExtractor):
} }
] ]
def _seconds2str(self, s):
return '%02d:%02d:%02d.%03d' % (s / 3600, (s % 3600) / 60, s % 60, (s % 1) * 1000)
def _debug_print(self, txt): def _debug_print(self, txt):
if self._downloader.params.get('verbose', False): if self._downloader.params.get('verbose', False):
self.to_screen('[debug] %s' % txt) self.to_screen('[debug] %s' % txt)
@ -168,8 +165,8 @@ class NRKTVIE(InfoExtractor):
for pos, p in enumerate(ps): for pos, p in enumerate(ps):
begin = parse_duration(p.get('begin')) begin = parse_duration(p.get('begin'))
duration = parse_duration(p.get('dur')) duration = parse_duration(p.get('dur'))
starttime = self._seconds2str(begin) starttime = self._subtitles_timecode(begin)
endtime = self._seconds2str(begin + duration) endtime = self._subtitles_timecode(begin + duration)
srt += '%s\r\n%s --> %s\r\n%s\r\n\r\n' % (compat_str(pos), starttime, endtime, p.text) srt += '%s\r\n%s --> %s\r\n%s\r\n\r\n' % (compat_str(pos), starttime, endtime, p.text)
return {lang: [ return {lang: [
{'ext': 'ttml', 'url': url}, {'ext': 'ttml', 'url': url},

94
youtube_dl/extractor/orf.py
View File

@ -11,6 +11,11 @@ from ..utils import (
HEADRequest, HEADRequest,
unified_strdate, unified_strdate,
ExtractorError, ExtractorError,
strip_jsonp,
int_or_none,
float_or_none,
determine_ext,
remove_end,
) )
@ -197,3 +202,92 @@ class ORFFM4IE(InfoExtractor):
'description': data['subtitle'], 'description': data['subtitle'],
'entries': entries 'entries': entries
} }
class ORFIPTVIE(InfoExtractor):
IE_NAME = 'orf:iptv'
IE_DESC = 'iptv.ORF.at'
_VALID_URL = r'http://iptv\.orf\.at/(?:#/)?stories/(?P<id>\d+)'
_TEST = {
'url': 'http://iptv.orf.at/stories/2267952',
'md5': '26ffa4bab6dbce1eee78bbc7021016cd',
'info_dict': {
'id': '339775',
'ext': 'flv',
'title': 'Kreml-Kritiker Nawalny wieder frei',
'description': 'md5:6f24e7f546d364dacd0e616a9e409236',
'duration': 84.729,
'thumbnail': 're:^https?://.*\.jpg$',
'upload_date': '20150306',
},
}
def _real_extract(self, url):
story_id = self._match_id(url)
webpage = self._download_webpage(
'http://iptv.orf.at/stories/%s' % story_id, story_id)
video_id = self._search_regex(
r'data-video(?:id)?="(\d+)"', webpage, 'video id')
data = self._download_json(
'http://bits.orf.at/filehandler/static-api/json/current/data.json?file=%s' % video_id,
video_id)[0]
duration = float_or_none(data['duration'], 1000)
video = data['sources']['default']
load_balancer_url = video['loadBalancerUrl']
abr = int_or_none(video.get('audioBitrate'))
vbr = int_or_none(video.get('bitrate'))
fps = int_or_none(video.get('videoFps'))
width = int_or_none(video.get('videoWidth'))
height = int_or_none(video.get('videoHeight'))
thumbnail = video.get('preview')
rendition = self._download_json(
load_balancer_url, video_id, transform_source=strip_jsonp)
f = {
'abr': abr,
'vbr': vbr,
'fps': fps,
'width': width,
'height': height,
}
formats = []
for format_id, format_url in rendition['redirect'].items():
if format_id == 'rtmp':
ff = f.copy()
ff.update({
'url': format_url,
'format_id': format_id,
})
formats.append(ff)
elif determine_ext(format_url) == 'f4m':
formats.extend(self._extract_f4m_formats(
format_url, video_id, f4m_id=format_id))
elif determine_ext(format_url) == 'm3u8':
formats.extend(self._extract_m3u8_formats(
format_url, video_id, 'mp4', m3u8_id=format_id))
else:
continue
self._sort_formats(formats)
title = remove_end(self._og_search_title(webpage), ' - iptv.ORF.at')
description = self._og_search_description(webpage)
upload_date = unified_strdate(self._html_search_meta(
'dc.date', webpage, 'upload date'))
return {
'id': video_id,
'title': title,
'description': description,
'duration': duration,
'thumbnail': thumbnail,
'upload_date': upload_date,
'formats': formats,
}

90
youtube_dl/extractor/pladform.py Normal file
View File

@ -0,0 +1,90 @@
# coding: utf-8
from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import (
ExtractorError,
int_or_none,
xpath_text,
qualities,
)
class PladformIE(InfoExtractor):
_VALID_URL = r'''(?x)
https?://
(?:
(?:
out\.pladform\.ru/player|
static\.pladform\.ru/player\.swf
)
\?.*\bvideoid=|
video\.pladform\.ru/catalog/video/videoid/
)
(?P<id>\d+)
'''
_TESTS = [{
# http://muz-tv.ru/kinozal/view/7400/
'url': 'http://out.pladform.ru/player?pl=24822&videoid=100183293',
'md5': '61f37b575dd27f1bb2e1854777fe31f4',
'info_dict': {
'id': '100183293',
'ext': 'mp4',
'title': 'Тайны перевала Дятлова • Тайна перевала Дятлова 1 серия 2 часть',
'description': 'Документальный сериал-расследование одной из самых жутких тайн ХХ века',
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 694,
'age_limit': 0,
},
}, {
'url': 'http://static.pladform.ru/player.swf?pl=21469&videoid=100183293&vkcid=0',
'only_matching': True,
}, {
'url': 'http://video.pladform.ru/catalog/video/videoid/100183293/vkcid/0',
'only_matching': True,
}]
def _real_extract(self, url):
video_id = self._match_id(url)
video = self._download_xml(
'http://out.pladform.ru/getVideo?pl=1&videoid=%s' % video_id,
video_id)
if video.tag == 'error':
raise ExtractorError(
'%s returned error: %s' % (self.IE_NAME, video.text),
expected=True)
quality = qualities(('ld', 'sd', 'hd'))
formats = [{
'url': src.text,
'format_id': src.get('quality'),
'quality': quality(src.get('quality')),
} for src in video.findall('./src')]
self._sort_formats(formats)
webpage = self._download_webpage(
'http://video.pladform.ru/catalog/video/videoid/%s' % video_id,
video_id)
title = self._og_search_title(webpage, fatal=False) or xpath_text(
video, './/title', 'title', fatal=True)
description = self._search_regex(
r'</h3>\s*<p>([^<]+)</p>', webpage, 'description', fatal=False)
thumbnail = self._og_search_thumbnail(webpage) or xpath_text(
video, './/cover', 'cover')
duration = int_or_none(xpath_text(video, './/time', 'duration'))
age_limit = int_or_none(xpath_text(video, './/age18', 'age limit'))
return {
'id': video_id,
'title': title,
'description': description,
'thumbnail': thumbnail,
'duration': duration,
'age_limit': age_limit,
'formats': formats,
}

78
youtube_dl/extractor/playwire.py Normal file
View File

@ -0,0 +1,78 @@
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
xpath_text,
float_or_none,
int_or_none,
)
class PlaywireIE(InfoExtractor):
_VALID_URL = r'https?://(?:config|cdn)\.playwire\.com(?:/v2)?/(?P<publisher_id>\d+)/(?:videos/v2|embed|config)/(?P<id>\d+)'
_TESTS = [{
'url': 'http://config.playwire.com/14907/videos/v2/3353705/player.json',
'md5': 'e6398701e3595888125729eaa2329ed9',
'info_dict': {
'id': '3353705',
'ext': 'mp4',
'title': 'S04_RM_UCL_Rus',
'thumbnail': 're:^http://.*\.png$',
'duration': 145.94,
},
}, {
'url': 'http://cdn.playwire.com/11625/embed/85228.html',
'only_matching': True,
}, {
'url': 'http://config.playwire.com/12421/videos/v2/3389892/zeus.json',
'only_matching': True,
}, {
'url': 'http://cdn.playwire.com/v2/12342/config/1532636.json',
'only_matching': True,
}]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
publisher_id, video_id = mobj.group('publisher_id'), mobj.group('id')
player = self._download_json(
'http://config.playwire.com/%s/videos/v2/%s/zeus.json' % (publisher_id, video_id),
video_id)
title = player['settings']['title']
duration = float_or_none(player.get('duration'), 1000)
content = player['content']
thumbnail = content.get('poster')
src = content['media']['f4m']
f4m = self._download_xml(src, video_id)
base_url = xpath_text(f4m, './{http://ns.adobe.com/f4m/1.0}baseURL', 'base url', fatal=True)
formats = []
for media in f4m.findall('./{http://ns.adobe.com/f4m/1.0}media'):
media_url = media.get('url')
if not media_url:
continue
tbr = int_or_none(media.get('bitrate'))
width = int_or_none(media.get('width'))
height = int_or_none(media.get('height'))
f = {
'url': '%s/%s' % (base_url, media.attrib['url']),
'tbr': tbr,
'width': width,
'height': height,
}
if not (tbr or width or height):
f['quality'] = 1 if '-hd.' in media_url else 0
formats.append(f)
self._sort_formats(formats)
return {
'id': video_id,
'title': title,
'thumbnail': thumbnail,
'duration': duration,
'formats': formats,
}

41
youtube_dl/extractor/rtve.py
View File

@ -127,6 +127,47 @@ class RTVEALaCartaIE(InfoExtractor):
for s in subs) for s in subs)
class RTVEInfantilIE(InfoExtractor):
IE_NAME = 'rtve.es:infantil'
IE_DESC = 'RTVE infantil'
_VALID_URL = r'https?://(?:www\.)?rtve\.es/infantil/serie/(?P<show>[^/]*)/video/(?P<short_title>[^/]*)/(?P<id>[0-9]+)/'
_TESTS = [{
'url': 'http://www.rtve.es/infantil/serie/cleo/video/maneras-vivir/3040283/',
'md5': '915319587b33720b8e0357caaa6617e6',
'info_dict': {
'id': '3040283',
'ext': 'mp4',
'title': 'Maneras de vivir',
'thumbnail': 'http://www.rtve.es/resources/jpg/6/5/1426182947956.JPG',
'duration': 357.958,
},
}]
def _real_extract(self, url):
video_id = self._match_id(url)
info = self._download_json(
'http://www.rtve.es/api/videos/%s/config/alacarta_videos.json' % video_id,
video_id)['page']['items'][0]
webpage = self._download_webpage(url, video_id)
vidplayer_id = self._search_regex(
r' id="vidplayer([0-9]+)"', webpage, 'internal video ID')
png_url = 'http://www.rtve.es/ztnr/movil/thumbnail/default/videos/%s.png' % vidplayer_id
png = self._download_webpage(png_url, video_id, 'Downloading url information')
video_url = _decrypt_url(png)
return {
'id': video_id,
'ext': 'mp4',
'title': info['title'],
'url': video_url,
'thumbnail': info.get('image'),
'duration': float_or_none(info.get('duration'), scale=1000),
}
class RTVELiveIE(InfoExtractor): class RTVELiveIE(InfoExtractor):
IE_NAME = 'rtve.es:live' IE_NAME = 'rtve.es:live'
IE_DESC = 'RTVE.es live streams' IE_DESC = 'RTVE.es live streams'

7
youtube_dl/extractor/soundcloud.py
View File

@ -180,7 +180,7 @@ class SoundcloudIE(InfoExtractor):
'format_id': key, 'format_id': key,
'url': url, 'url': url,
'play_path': 'mp3:' + path, 'play_path': 'mp3:' + path,
'ext': ext, 'ext': 'flv',
'vcodec': 'none', 'vcodec': 'none',
}) })
@ -200,8 +200,9 @@ class SoundcloudIE(InfoExtractor):
if f['format_id'].startswith('rtmp'): if f['format_id'].startswith('rtmp'):
f['protocol'] = 'rtmp' f['protocol'] = 'rtmp'
self._sort_formats(formats) self._check_formats(formats, track_id)
result['formats'] = formats self._sort_formats(formats)
result['formats'] = formats
return result return result

58
youtube_dl/extractor/ssa.py Normal file
View File

@ -0,0 +1,58 @@
from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import (
unescapeHTML,
parse_duration,
)
class SSAIE(InfoExtractor):
_VALID_URL = r'http://ssa\.nls\.uk/film/(?P<id>\d+)'
_TEST = {
'url': 'http://ssa.nls.uk/film/3561',
'info_dict': {
'id': '3561',
'ext': 'flv',
'title': 'SHETLAND WOOL',
'description': 'md5:c5afca6871ad59b4271e7704fe50ab04',
'duration': 900,
'thumbnail': 're:^https?://.*\.jpg$',
},
'params': {
# rtmp download
'skip_download': True,
},
}
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
streamer = self._search_regex(
r"'streamer'\s*,\S*'(rtmp[^']+)'", webpage, 'streamer')
play_path = self._search_regex(
r"'file'\s*,\s*'([^']+)'", webpage, 'file').rpartition('.')[0]
def search_field(field_name, fatal=False):
return self._search_regex(
r'<span\s+class="field_title">%s:</span>\s*<span\s+class="field_content">([^<]+)</span>' % field_name,
webpage, 'title', fatal=fatal)
title = unescapeHTML(search_field('Title', fatal=True)).strip('()[]')
description = unescapeHTML(search_field('Description'))
duration = parse_duration(search_field('Running time'))
thumbnail = self._search_regex(
r"'image'\s*,\s*'([^']+)'", webpage, 'thumbnails', fatal=False)
return {
'id': video_id,
'url': streamer,
'play_path': play_path,
'ext': 'flv',
'title': title,
'description': description,
'duration': duration,
'thumbnail': thumbnail,
}

6
youtube_dl/extractor/teamcoco.py
View File

@ -53,10 +53,10 @@ class TeamcocoIE(InfoExtractor):
embed = self._download_webpage( embed = self._download_webpage(
embed_url, video_id, 'Downloading embed page') embed_url, video_id, 'Downloading embed page')
encoded_data = self._search_regex( player_data = self._parse_json(self._search_regex(
r'"preload"\s*:\s*"([^"]+)"', embed, 'encoded data') r'Y\.Ginger\.Module\.Player\((\{.*?\})\);', embed, 'player data'), video_id)
data = self._parse_json( data = self._parse_json(
base64.b64decode(encoded_data.encode('ascii')).decode('utf-8'), video_id) base64.b64decode(player_data['preload'].encode('ascii')).decode('utf-8'), video_id)
formats = [] formats = []
get_quality = qualities(['500k', '480p', '1000k', '720p', '1080p']) get_quality = qualities(['500k', '480p', '1000k', '720p', '1080p'])

3
youtube_dl/extractor/tvplay.py
View File

@ -16,6 +16,7 @@ class TVPlayIE(InfoExtractor):
_VALID_URL = r'''(?x)http://(?:www\.)? _VALID_URL = r'''(?x)http://(?:www\.)?
(?:tvplay\.lv/parraides| (?:tvplay\.lv/parraides|
tv3play\.lt/programos| tv3play\.lt/programos|
play\.tv3\.lt/programos|
tv3play\.ee/sisu| tv3play\.ee/sisu|
tv3play\.se/program| tv3play\.se/program|
tv6play\.se/program| tv6play\.se/program|
@ -45,7 +46,7 @@ class TVPlayIE(InfoExtractor):
}, },
}, },
{ {
'url': 'http://www.tv3play.lt/programos/moterys-meluoja-geriau/409229?autostart=true', 'url': 'http://play.tv3.lt/programos/moterys-meluoja-geriau/409229?autostart=true',
'info_dict': { 'info_dict': {
'id': '409229', 'id': '409229',
'ext': 'flv', 'ext': 'flv',

25
youtube_dl/extractor/twitch.py
View File

@ -23,6 +23,7 @@ class TwitchBaseIE(InfoExtractor):
_API_BASE = 'https://api.twitch.tv' _API_BASE = 'https://api.twitch.tv'
_USHER_BASE = 'http://usher.twitch.tv' _USHER_BASE = 'http://usher.twitch.tv'
_LOGIN_URL = 'https://secure.twitch.tv/user/login' _LOGIN_URL = 'https://secure.twitch.tv/user/login'
_NETRC_MACHINE = 'twitch'
def _handle_error(self, response): def _handle_error(self, response):
if not isinstance(response, dict): if not isinstance(response, dict):
@ -84,6 +85,14 @@ class TwitchBaseIE(InfoExtractor):
raise ExtractorError( raise ExtractorError(
'Unable to login: %s' % m.group('msg').strip(), expected=True) 'Unable to login: %s' % m.group('msg').strip(), expected=True)
def _prefer_source(self, formats):
try:
source = next(f for f in formats if f['format_id'] == 'Source')
source['preference'] = 10
except StopIteration:
pass # No Source stream present
self._sort_formats(formats)
class TwitchItemBaseIE(TwitchBaseIE): class TwitchItemBaseIE(TwitchBaseIE):
def _download_info(self, item, item_id): def _download_info(self, item, item_id):
@ -208,6 +217,7 @@ class TwitchVodIE(TwitchItemBaseIE):
'%s/vod/%s?nauth=%s&nauthsig=%s' '%s/vod/%s?nauth=%s&nauthsig=%s'
% (self._USHER_BASE, item_id, access_token['token'], access_token['sig']), % (self._USHER_BASE, item_id, access_token['token'], access_token['sig']),
item_id, 'mp4') item_id, 'mp4')
self._prefer_source(formats)
info['formats'] = formats info['formats'] = formats
return info return info
@ -348,21 +358,14 @@ class TwitchStreamIE(TwitchBaseIE):
'p': random.randint(1000000, 10000000), 'p': random.randint(1000000, 10000000),
'player': 'twitchweb', 'player': 'twitchweb',
'segment_preference': '4', 'segment_preference': '4',
'sig': access_token['sig'], 'sig': access_token['sig'].encode('utf-8'),
'token': access_token['token'], 'token': access_token['token'].encode('utf-8'),
} }
formats = self._extract_m3u8_formats( formats = self._extract_m3u8_formats(
'%s/api/channel/hls/%s.m3u8?%s' '%s/api/channel/hls/%s.m3u8?%s'
% (self._USHER_BASE, channel_id, compat_urllib_parse.urlencode(query).encode('utf-8')), % (self._USHER_BASE, channel_id, compat_urllib_parse.urlencode(query)),
channel_id, 'mp4') channel_id, 'mp4')
self._prefer_source(formats)
# prefer the 'source' stream, the others are limited to 30 fps
def _sort_source(f):
if f.get('m3u8_media') is not None and f['m3u8_media'].get('NAME') == 'Source':
return 1
return 0
formats = sorted(formats, key=_sort_source)
view_count = stream.get('viewers') view_count = stream.get('viewers')
timestamp = parse_iso8601(stream.get('created_at')) timestamp = parse_iso8601(stream.get('created_at'))

15
youtube_dl/extractor/vidme.py
View File

@ -1,7 +1,5 @@
from __future__ import unicode_literals from __future__ import unicode_literals
import re
from .common import InfoExtractor from .common import InfoExtractor
from ..utils import ( from ..utils import (
int_or_none, int_or_none,
@ -28,12 +26,11 @@ class VidmeIE(InfoExtractor):
} }
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) video_id = self._match_id(url)
video_id = mobj.group('id')
webpage = self._download_webpage(url, video_id) webpage = self._download_webpage(url, video_id)
video_url = self._html_search_regex(r'<source src="([^"]+)"', webpage, 'video URL') video_url = self._html_search_regex(
r'<source src="([^"]+)"', webpage, 'video URL')
title = self._og_search_title(webpage) title = self._og_search_title(webpage)
description = self._og_search_description(webpage, default='') description = self._og_search_description(webpage, default='')
@ -44,13 +41,10 @@ class VidmeIE(InfoExtractor):
duration = float_or_none(self._html_search_regex( duration = float_or_none(self._html_search_regex(
r'data-duration="([^"]+)"', webpage, 'duration', fatal=False)) r'data-duration="([^"]+)"', webpage, 'duration', fatal=False))
view_count = str_to_int(self._html_search_regex( view_count = str_to_int(self._html_search_regex(
r'<span class="video_views">\s*([\d,\.]+)\s*plays?', webpage, 'view count', fatal=False)) r'<(?:li|span) class="video_views">\s*([\d,\.]+)\s*plays?', webpage, 'view count', fatal=False))
like_count = str_to_int(self._html_search_regex( like_count = str_to_int(self._html_search_regex(
r'class="score js-video-vote-score"[^>]+data-score="([\d,\.\s]+)">', r'class="score js-video-vote-score"[^>]+data-score="([\d,\.\s]+)">',
webpage, 'like count', fatal=False)) webpage, 'like count', fatal=False))
comment_count = str_to_int(self._html_search_regex(
r'class="js-comment-count"[^>]+data-count="([\d,\.\s]+)">',
webpage, 'comment count', fatal=False))
return { return {
'id': video_id, 'id': video_id,
@ -64,5 +58,4 @@ class VidmeIE(InfoExtractor):
'duration': duration, 'duration': duration,
'view_count': view_count, 'view_count': view_count,
'like_count': like_count, 'like_count': like_count,
'comment_count': comment_count,
} }

129
youtube_dl/extractor/viewster.py Normal file
View File

@ -0,0 +1,129 @@
from __future__ import unicode_literals
from .common import InfoExtractor
from ..compat import compat_urllib_request
class ViewsterIE(InfoExtractor):
_VALID_URL = r'http://(?:www\.)?viewster\.com/movie/(?P<id>\d+-\d+-\d+)'
_TESTS = [{
# movielink, paymethod=fre
'url': 'http://www.viewster.com/movie/1293-19341-000/hout-wood/',
'playlist': [{
'md5': '8f9d94b282d80c42b378dffdbb11caf3',
'info_dict': {
'id': '1293-19341-000-movie',
'ext': 'flv',
'title': "'Hout' (Wood) - Movie",
},
}],
'info_dict': {
'id': '1293-19341-000',
'title': "'Hout' (Wood)",
'description': 'md5:925733185a9242ef96f436937683f33b',
}
}, {
# movielink, paymethod=adv
'url': 'http://www.viewster.com/movie/1140-11855-000/the-listening-project/',
'playlist': [{
'md5': '77a005453ca7396cbe3d35c9bea30aef',
'info_dict': {
'id': '1140-11855-000-movie',
'ext': 'flv',
'title': "THE LISTENING PROJECT - Movie",
},
}],
'info_dict': {
'id': '1140-11855-000',
'title': "THE LISTENING PROJECT",
'description': 'md5:714421ae9957e112e672551094bf3b08',
}
}, {
# direct links, no movielink
'url': 'http://www.viewster.com/movie/1198-56411-000/sinister/',
'playlist': [{
'md5': '0307b7eac6bfb21ab0577a71f6eebd8f',
'info_dict': {
'id': '1198-56411-000-trailer',
'ext': 'mp4',
'title': "Sinister - Trailer",
},
}, {
'md5': '80b9ee3ad69fb368f104cb5d9732ae95',
'info_dict': {
'id': '1198-56411-000-behind-scenes',
'ext': 'mp4',
'title': "Sinister - Behind Scenes",
},
}, {
'md5': '3b3ea897ecaa91fca57a8a94ac1b15c5',
'info_dict': {
'id': '1198-56411-000-scene-from-movie',
'ext': 'mp4',
'title': "Sinister - Scene from movie",
},
}],
'info_dict': {
'id': '1198-56411-000',
'title': "Sinister",
'description': 'md5:014c40b0488848de9683566a42e33372',
}
}]
_ACCEPT_HEADER = 'application/json, text/javascript, */*; q=0.01'
def _real_extract(self, url):
video_id = self._match_id(url)
request = compat_urllib_request.Request(
'http://api.live.viewster.com/api/v1/movie/%s' % video_id)
request.add_header('Accept', self._ACCEPT_HEADER)
movie = self._download_json(
request, video_id, 'Downloading movie metadata JSON')
title = movie.get('title') or movie['original_title']
description = movie.get('synopsis')
thumbnail = movie.get('large_artwork') or movie.get('artwork')
entries = []
for clip in movie['play_list']:
entry = None
# movielink api
link_request = clip.get('link_request')
if link_request:
request = compat_urllib_request.Request(
'http://api.live.viewster.com/api/v1/movielink?movieid=%(movieid)s&action=%(action)s&paymethod=%(paymethod)s&price=%(price)s&currency=%(currency)s&language=%(language)s&subtitlelanguage=%(subtitlelanguage)s&ischromecast=%(ischromecast)s'
% link_request)
request.add_header('Accept', self._ACCEPT_HEADER)
movie_link = self._download_json(
request, video_id, 'Downloading movie link JSON', fatal=False)
if movie_link:
formats = self._extract_f4m_formats(
movie_link['url'] + '&hdcore=3.2.0&plugin=flowplayer-3.2.0.1', video_id)
self._sort_formats(formats)
entry = {
'formats': formats,
}
# direct link
clip_url = clip.get('clip_data', {}).get('url')
if clip_url:
entry = {
'url': clip_url,
'ext': 'mp4',
}
if entry:
entry.update({
'id': '%s-%s' % (video_id, clip['canonical_title']),
'title': '%s - %s' % (title, clip['title']),
})
entries.append(entry)
playlist = self.playlist_result(entries, video_id, title, description)
playlist['thumbnail'] = thumbnail
return playlist

76
youtube_dl/extractor/vimeo.py
View File

@ -4,7 +4,6 @@ from __future__ import unicode_literals
import json import json
import re import re
import itertools import itertools
import hashlib
from .common import InfoExtractor from .common import InfoExtractor
from ..compat import ( from ..compat import (
@ -20,6 +19,7 @@ from ..utils import (
RegexNotFoundError, RegexNotFoundError,
smuggle_url, smuggle_url,
std_headers, std_headers,
unified_strdate,
unsmuggle_url, unsmuggle_url,
urlencode_postdata, urlencode_postdata,
) )
@ -38,7 +38,7 @@ class VimeoBaseInfoExtractor(InfoExtractor):
self.report_login() self.report_login()
login_url = 'https://vimeo.com/log_in' login_url = 'https://vimeo.com/log_in'
webpage = self._download_webpage(login_url, None, False) webpage = self._download_webpage(login_url, None, False)
token = self._search_regex(r'xsrft: \'(.*?)\'', webpage, 'login token') token = self._search_regex(r'xsrft = \'(.*?)\'', webpage, 'login token')
data = urlencode_postdata({ data = urlencode_postdata({
'email': username, 'email': username,
'password': password, 'password': password,
@ -140,6 +140,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
'description': 'md5:8678b246399b070816b12313e8b4eb5c', 'description': 'md5:8678b246399b070816b12313e8b4eb5c',
'uploader_id': 'atencio', 'uploader_id': 'atencio',
'uploader': 'Peter Atencio', 'uploader': 'Peter Atencio',
'upload_date': '20130927',
'duration': 187, 'duration': 187,
}, },
}, },
@ -176,17 +177,15 @@ class VimeoIE(VimeoBaseInfoExtractor):
password = self._downloader.params.get('videopassword', None) password = self._downloader.params.get('videopassword', None)
if password is None: if password is None:
raise ExtractorError('This video is protected by a password, use the --video-password option', expected=True) raise ExtractorError('This video is protected by a password, use the --video-password option', expected=True)
token = self._search_regex(r'xsrft: \'(.*?)\'', webpage, 'login token') token = self._search_regex(r'xsrft = \'(.*?)\'', webpage, 'login token')
data = compat_urllib_parse.urlencode({ data = urlencode_postdata({
'password': password, 'password': password,
'token': token, 'token': token,
}) })
# I didn't manage to use the password with https if url.startswith('http://'):
if url.startswith('https'): # vimeo only supports https now, but the user can give an http url
pass_url = url.replace('https', 'http') url = url.replace('http://', 'https://')
else: password_request = compat_urllib_request.Request(url + '/password', data)
pass_url = url
password_request = compat_urllib_request.Request(pass_url + '/password', data)
password_request.add_header('Content-Type', 'application/x-www-form-urlencoded') password_request.add_header('Content-Type', 'application/x-www-form-urlencoded')
password_request.add_header('Cookie', 'xsrft=%s' % token) password_request.add_header('Cookie', 'xsrft=%s' % token)
return self._download_webpage( return self._download_webpage(
@ -223,12 +222,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
video_id = mobj.group('id') video_id = mobj.group('id')
orig_url = url orig_url = url
if mobj.group('pro') or mobj.group('player'): if mobj.group('pro') or mobj.group('player'):
url = 'http://player.vimeo.com/video/' + video_id url = 'https://player.vimeo.com/video/' + video_id
password = self._downloader.params.get('videopassword', None)
if password:
headers['Cookie'] = '%s_password=%s' % (
video_id, hashlib.md5(password.encode('utf-8')).hexdigest())
# Retrieve video webpage to extract further information # Retrieve video webpage to extract further information
request = compat_urllib_request.Request(url, None, headers) request = compat_urllib_request.Request(url, None, headers)
@ -323,9 +317,9 @@ class VimeoIE(VimeoBaseInfoExtractor):
# Extract upload date # Extract upload date
video_upload_date = None video_upload_date = None
mobj = re.search(r'<meta itemprop="dateCreated" content="(\d{4})-(\d{2})-(\d{2})T', webpage) mobj = re.search(r'<time[^>]+datetime="([^"]+)"', webpage)
if mobj is not None: if mobj is not None:
video_upload_date = mobj.group(1) + mobj.group(2) + mobj.group(3) video_upload_date = unified_strdate(mobj.group(1))
try: try:
view_count = int(self._search_regex(r'UserPlays:(\d+)', webpage, 'view count')) view_count = int(self._search_regex(r'UserPlays:(\d+)', webpage, 'view count'))
@ -379,7 +373,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
for tt in text_tracks: for tt in text_tracks:
subtitles[tt['lang']] = [{ subtitles[tt['lang']] = [{
'ext': 'vtt', 'ext': 'vtt',
'url': 'http://vimeo.com' + tt['url'], 'url': 'https://vimeo.com' + tt['url'],
}] }]
return { return {
@ -402,11 +396,11 @@ class VimeoIE(VimeoBaseInfoExtractor):
class VimeoChannelIE(InfoExtractor): class VimeoChannelIE(InfoExtractor):
IE_NAME = 'vimeo:channel' IE_NAME = 'vimeo:channel'
_VALID_URL = r'https?://vimeo\.com/channels/(?P<id>[^/?#]+)/?(?:$|[?#])' _VALID_URL = r'https://vimeo\.com/channels/(?P<id>[^/?#]+)/?(?:$|[?#])'
_MORE_PAGES_INDICATOR = r'<a.+?rel="next"' _MORE_PAGES_INDICATOR = r'<a.+?rel="next"'
_TITLE_RE = r'<link rel="alternate"[^>]+?title="(.*?)"' _TITLE_RE = r'<link rel="alternate"[^>]+?title="(.*?)"'
_TESTS = [{ _TESTS = [{
'url': 'http://vimeo.com/channels/tributes', 'url': 'https://vimeo.com/channels/tributes',
'info_dict': { 'info_dict': {
'id': 'tributes', 'id': 'tributes',
'title': 'Vimeo Tributes', 'title': 'Vimeo Tributes',
@ -435,10 +429,10 @@ class VimeoChannelIE(InfoExtractor):
name="([^"]+)"\s+ name="([^"]+)"\s+
value="([^"]*)" value="([^"]*)"
''', login_form)) ''', login_form))
token = self._search_regex(r'xsrft: \'(.*?)\'', webpage, 'login token') token = self._search_regex(r'xsrft = \'(.*?)\'', webpage, 'login token')
fields['token'] = token fields['token'] = token
fields['password'] = password fields['password'] = password
post = compat_urllib_parse.urlencode(fields) post = urlencode_postdata(fields)
password_path = self._search_regex( password_path = self._search_regex(
r'action="([^"]+)"', login_form, 'password URL') r'action="([^"]+)"', login_form, 'password URL')
password_url = compat_urlparse.urljoin(page_url, password_path) password_url = compat_urlparse.urljoin(page_url, password_path)
@ -465,7 +459,7 @@ class VimeoChannelIE(InfoExtractor):
if re.search(self._MORE_PAGES_INDICATOR, webpage, re.DOTALL) is None: if re.search(self._MORE_PAGES_INDICATOR, webpage, re.DOTALL) is None:
break break
entries = [self.url_result('http://vimeo.com/%s' % video_id, 'Vimeo') entries = [self.url_result('https://vimeo.com/%s' % video_id, 'Vimeo')
for video_id in video_ids] for video_id in video_ids]
return {'_type': 'playlist', return {'_type': 'playlist',
'id': list_id, 'id': list_id,
@ -476,15 +470,15 @@ class VimeoChannelIE(InfoExtractor):
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
channel_id = mobj.group('id') channel_id = mobj.group('id')
return self._extract_videos(channel_id, 'http://vimeo.com/channels/%s' % channel_id) return self._extract_videos(channel_id, 'https://vimeo.com/channels/%s' % channel_id)
class VimeoUserIE(VimeoChannelIE): class VimeoUserIE(VimeoChannelIE):
IE_NAME = 'vimeo:user' IE_NAME = 'vimeo:user'
_VALID_URL = r'https?://vimeo\.com/(?![0-9]+(?:$|[?#/]))(?P<name>[^/]+)(?:/videos|[#?]|$)' _VALID_URL = r'https://vimeo\.com/(?![0-9]+(?:$|[?#/]))(?P<name>[^/]+)(?:/videos|[#?]|$)'
_TITLE_RE = r'<a[^>]+?class="user">([^<>]+?)</a>' _TITLE_RE = r'<a[^>]+?class="user">([^<>]+?)</a>'
_TESTS = [{ _TESTS = [{
'url': 'http://vimeo.com/nkistudio/videos', 'url': 'https://vimeo.com/nkistudio/videos',
'info_dict': { 'info_dict': {
'title': 'Nki', 'title': 'Nki',
'id': 'nkistudio', 'id': 'nkistudio',
@ -495,15 +489,15 @@ class VimeoUserIE(VimeoChannelIE):
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
name = mobj.group('name') name = mobj.group('name')
return self._extract_videos(name, 'http://vimeo.com/%s' % name) return self._extract_videos(name, 'https://vimeo.com/%s' % name)
class VimeoAlbumIE(VimeoChannelIE): class VimeoAlbumIE(VimeoChannelIE):
IE_NAME = 'vimeo:album' IE_NAME = 'vimeo:album'
_VALID_URL = r'https?://vimeo\.com/album/(?P<id>\d+)' _VALID_URL = r'https://vimeo\.com/album/(?P<id>\d+)'
_TITLE_RE = r'<header id="page_header">\n\s*<h1>(.*?)</h1>' _TITLE_RE = r'<header id="page_header">\n\s*<h1>(.*?)</h1>'
_TESTS = [{ _TESTS = [{
'url': 'http://vimeo.com/album/2632481', 'url': 'https://vimeo.com/album/2632481',
'info_dict': { 'info_dict': {
'id': '2632481', 'id': '2632481',
'title': 'Staff Favorites: November 2013', 'title': 'Staff Favorites: November 2013',
@ -527,14 +521,14 @@ class VimeoAlbumIE(VimeoChannelIE):
def _real_extract(self, url): def _real_extract(self, url):
album_id = self._match_id(url) album_id = self._match_id(url)
return self._extract_videos(album_id, 'http://vimeo.com/album/%s' % album_id) return self._extract_videos(album_id, 'https://vimeo.com/album/%s' % album_id)
class VimeoGroupsIE(VimeoAlbumIE): class VimeoGroupsIE(VimeoAlbumIE):
IE_NAME = 'vimeo:group' IE_NAME = 'vimeo:group'
_VALID_URL = r'(?:https?://)?vimeo\.com/groups/(?P<name>[^/]+)' _VALID_URL = r'https://vimeo\.com/groups/(?P<name>[^/]+)'
_TESTS = [{ _TESTS = [{
'url': 'http://vimeo.com/groups/rolexawards', 'url': 'https://vimeo.com/groups/rolexawards',
'info_dict': { 'info_dict': {
'id': 'rolexawards', 'id': 'rolexawards',
'title': 'Rolex Awards for Enterprise', 'title': 'Rolex Awards for Enterprise',
@ -548,13 +542,13 @@ class VimeoGroupsIE(VimeoAlbumIE):
def _real_extract(self, url): def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url) mobj = re.match(self._VALID_URL, url)
name = mobj.group('name') name = mobj.group('name')
return self._extract_videos(name, 'http://vimeo.com/groups/%s' % name) return self._extract_videos(name, 'https://vimeo.com/groups/%s' % name)
class VimeoReviewIE(InfoExtractor): class VimeoReviewIE(InfoExtractor):
IE_NAME = 'vimeo:review' IE_NAME = 'vimeo:review'
IE_DESC = 'Review pages on vimeo' IE_DESC = 'Review pages on vimeo'
_VALID_URL = r'https?://vimeo\.com/[^/]+/review/(?P<id>[^/]+)' _VALID_URL = r'https://vimeo\.com/[^/]+/review/(?P<id>[^/]+)'
_TESTS = [{ _TESTS = [{
'url': 'https://vimeo.com/user21297594/review/75524534/3c257a1b5d', 'url': 'https://vimeo.com/user21297594/review/75524534/3c257a1b5d',
'md5': 'c507a72f780cacc12b2248bb4006d253', 'md5': 'c507a72f780cacc12b2248bb4006d253',
@ -566,7 +560,7 @@ class VimeoReviewIE(InfoExtractor):
} }
}, { }, {
'note': 'video player needs Referer', 'note': 'video player needs Referer',
'url': 'http://vimeo.com/user22258446/review/91613211/13f927e053', 'url': 'https://vimeo.com/user22258446/review/91613211/13f927e053',
'md5': '6295fdab8f4bf6a002d058b2c6dce276', 'md5': '6295fdab8f4bf6a002d058b2c6dce276',
'info_dict': { 'info_dict': {
'id': '91613211', 'id': '91613211',
@ -588,11 +582,11 @@ class VimeoReviewIE(InfoExtractor):
class VimeoWatchLaterIE(VimeoBaseInfoExtractor, VimeoChannelIE): class VimeoWatchLaterIE(VimeoBaseInfoExtractor, VimeoChannelIE):
IE_NAME = 'vimeo:watchlater' IE_NAME = 'vimeo:watchlater'
IE_DESC = 'Vimeo watch later list, "vimeowatchlater" keyword (requires authentication)' IE_DESC = 'Vimeo watch later list, "vimeowatchlater" keyword (requires authentication)'
_VALID_URL = r'https?://vimeo\.com/home/watchlater|:vimeowatchlater' _VALID_URL = r'https://vimeo\.com/home/watchlater|:vimeowatchlater'
_LOGIN_REQUIRED = True _LOGIN_REQUIRED = True
_TITLE_RE = r'href="/home/watchlater".*?>(.*?)<' _TITLE_RE = r'href="/home/watchlater".*?>(.*?)<'
_TESTS = [{ _TESTS = [{
'url': 'http://vimeo.com/home/watchlater', 'url': 'https://vimeo.com/home/watchlater',
'only_matching': True, 'only_matching': True,
}] }]
@ -612,7 +606,7 @@ class VimeoWatchLaterIE(VimeoBaseInfoExtractor, VimeoChannelIE):
class VimeoLikesIE(InfoExtractor): class VimeoLikesIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?vimeo\.com/user(?P<id>[0-9]+)/likes/?(?:$|[?#]|sort:)' _VALID_URL = r'https://(?:www\.)?vimeo\.com/user(?P<id>[0-9]+)/likes/?(?:$|[?#]|sort:)'
IE_NAME = 'vimeo:likes' IE_NAME = 'vimeo:likes'
IE_DESC = 'Vimeo user likes' IE_DESC = 'Vimeo user likes'
_TEST = { _TEST = {
@ -640,8 +634,8 @@ class VimeoLikesIE(InfoExtractor):
description = self._html_search_meta('description', webpage) description = self._html_search_meta('description', webpage)
def _get_page(idx): def _get_page(idx):
page_url = '%s//vimeo.com/user%s/likes/page:%d/sort:date' % ( page_url = 'https://vimeo.com/user%s/likes/page:%d/sort:date' % (
self.http_scheme(), user_id, idx + 1) user_id, idx + 1)
webpage = self._download_webpage( webpage = self._download_webpage(
page_url, user_id, page_url, user_id,
note='Downloading page %d/%d' % (idx + 1, page_count)) note='Downloading page %d/%d' % (idx + 1, page_count))

2
youtube_dl/extractor/vk.py
View File

@ -31,7 +31,7 @@ class VKIE(InfoExtractor):
'id': '162222515', 'id': '162222515',
'ext': 'flv', 'ext': 'flv',
'title': 'ProtivoGunz - Хуёвая песня', 'title': 'ProtivoGunz - Хуёвая песня',
'uploader': 're:Noize MC.*', 'uploader': 're:(?:Noize MC|Alexander Ilyashenko).*',
'duration': 195, 'duration': 195,
'upload_date': '20120212', 'upload_date': '20120212',
}, },

37
youtube_dl/extractor/yam.py
View File

@ -8,6 +8,7 @@ from ..compat import compat_urlparse
from ..utils import ( from ..utils import (
float_or_none, float_or_none,
month_by_abbreviation, month_by_abbreviation,
ExtractorError,
) )
@ -28,23 +29,45 @@ class YamIE(InfoExtractor):
} }
}, { }, {
# An external video hosted on YouTube # An external video hosted on YouTube
'url': 'http://mymedia.yam.com/m/3598173', 'url': 'http://mymedia.yam.com/m/3599430',
'md5': '0238ceec479c654e8c2f1223755bf3e9', 'md5': '03127cf10d8f35d120a9e8e52e3b17c6',
'info_dict': { 'info_dict': {
'id': 'pJ2Deys283c', 'id': 'CNpEoQlrIgA',
'ext': 'mp4', 'ext': 'mp4',
'upload_date': '20150202', 'upload_date': '20150306',
'uploader': '新莊社大瑜伽社', 'uploader': '新莊社大瑜伽社',
'description': 'md5:f5cc72f0baf259a70fb731654b0d2eff', 'description': 'md5:11e2e405311633ace874f2e6226c8b17',
'uploader_id': '2323agoy', 'uploader_id': '2323agoy',
'title': '外婆的澎湖灣KTV-潘安邦', 'title': '20090412陽明山二子坪-1',
} },
'skip': 'Video does not exist',
}, {
'url': 'http://mymedia.yam.com/m/3598173',
'info_dict': {
'id': '3598173',
'ext': 'mp4',
},
'skip': 'cause Yam system error',
}, {
'url': 'http://mymedia.yam.com/m/3599437',
'info_dict': {
'id': '3599437',
'ext': 'mp4',
},
'skip': 'invalid YouTube URL',
}] }]
def _real_extract(self, url): def _real_extract(self, url):
video_id = self._match_id(url) video_id = self._match_id(url)
page = self._download_webpage(url, video_id) page = self._download_webpage(url, video_id)
# Check for errors
system_msg = self._html_search_regex(
r'系統訊息(?:<br>|\n|\r)*([^<>]+)<br>', page, 'system message',
default=None)
if system_msg:
raise ExtractorError(system_msg, expected=True)
# Is it hosted externally on YouTube? # Is it hosted externally on YouTube?
youtube_url = self._html_search_regex( youtube_url = self._html_search_regex(
r'<embed src="(http://www.youtube.com/[^"]+)"', r'<embed src="(http://www.youtube.com/[^"]+)"',

127
youtube_dl/extractor/yandexmusic.py Normal file
View File

@ -0,0 +1,127 @@
# coding=utf-8
from __future__ import unicode_literals
import re
import hashlib
from .common import InfoExtractor
from ..compat import compat_str
from ..utils import (
int_or_none,
float_or_none,
)
class YandexMusicBaseIE(InfoExtractor):
def _get_track_url(self, storage_dir, track_id):
data = self._download_json(
'http://music.yandex.ru/api/v1.5/handlers/api-jsonp.jsx?action=getTrackSrc&p=download-info/%s'
% storage_dir,
track_id, 'Downloading track location JSON')
key = hashlib.md5(('XGRlBW9FXlekgbPrRHuSiA' + data['path'][1:] + data['s']).encode('utf-8')).hexdigest()
storage = storage_dir.split('.')
return ('http://%s/get-mp3/%s/%s?track-id=%s&from=service-10-track&similarities-experiment=default'
% (data['host'], key, data['ts'] + data['path'], storage[1]))
def _get_track_info(self, track):
return {
'id': track['id'],
'ext': 'mp3',
'url': self._get_track_url(track['storageDir'], track['id']),
'title': '%s - %s' % (track['artists'][0]['name'], track['title']),
'filesize': int_or_none(track.get('fileSize')),
'duration': float_or_none(track.get('durationMs'), 1000),
}
class YandexMusicTrackIE(YandexMusicBaseIE):
IE_NAME = 'yandexmusic:track'
IE_DESC = 'Яндекс.Музыка - Трек'
_VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/album/(?P<album_id>\d+)/track/(?P<id>\d+)'
_TEST = {
'url': 'http://music.yandex.ru/album/540508/track/4878838',
'md5': 'f496818aa2f60b6c0062980d2e00dc20',
'info_dict': {
'id': '4878838',
'ext': 'mp3',
'title': 'Carlo Ambrosio - Gypsy Eyes 1',
'filesize': 4628061,
'duration': 193.04,
}
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
album_id, track_id = mobj.group('album_id'), mobj.group('id')
track = self._download_json(
'http://music.yandex.ru/handlers/track.jsx?track=%s:%s' % (track_id, album_id),
track_id, 'Downloading track JSON')['track']
return self._get_track_info(track)
class YandexMusicAlbumIE(YandexMusicBaseIE):
IE_NAME = 'yandexmusic:album'
IE_DESC = 'Яндекс.Музыка - Альбом'
_VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/album/(?P<id>\d+)/?(\?|$)'
_TEST = {
'url': 'http://music.yandex.ru/album/540508',
'info_dict': {
'id': '540508',
'title': 'Carlo Ambrosio - Gypsy Soul (2009)',
},
'playlist_count': 50,
}
def _real_extract(self, url):
album_id = self._match_id(url)
album = self._download_json(
'http://music.yandex.ru/handlers/album.jsx?album=%s' % album_id,
album_id, 'Downloading album JSON')
entries = [self._get_track_info(track) for track in album['volumes'][0]]
title = '%s - %s' % (album['artists'][0]['name'], album['title'])
year = album.get('year')
if year:
title += ' (%s)' % year
return self.playlist_result(entries, compat_str(album['id']), title)
class YandexMusicPlaylistIE(YandexMusicBaseIE):
IE_NAME = 'yandexmusic:playlist'
IE_DESC = 'Яндекс.Музыка - Плейлист'
_VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/users/[^/]+/playlists/(?P<id>\d+)'
_TEST = {
'url': 'http://music.yandex.ru/users/music.partners/playlists/1245',
'info_dict': {
'id': '1245',
'title': 'Что слушают Enter Shikari',
'description': 'md5:3b9f27b0efbe53f2ee1e844d07155cc9',
},
'playlist_count': 6,
}
def _real_extract(self, url):
playlist_id = self._match_id(url)
webpage = self._download_webpage(url, playlist_id)
playlist = self._parse_json(
self._search_regex(
r'var\s+Mu\s*=\s*({.+?});\s*</script>', webpage, 'player'),
playlist_id)['pageData']['playlist']
entries = [self._get_track_info(track) for track in playlist['tracks']]
return self.playlist_result(
entries, compat_str(playlist_id),
playlist['title'], playlist.get('description'))

3
youtube_dl/extractor/youporn.py
View File

@ -47,7 +47,8 @@ class YouPornIE(InfoExtractor):
# Get JSON parameters # Get JSON parameters
json_params = self._search_regex( json_params = self._search_regex(
r'var currentVideo = new Video\((.*)\)[,;]', [r'var\s+videoJa?son\s*=\s*({.+?});',
r'var\s+currentVideo\s*=\s*new\s+Video\((.+?)\)[,;]'],
webpage, 'JSON parameters') webpage, 'JSON parameters')
try: try:
params = json.loads(json_params) params = json.loads(json_params)

2
youtube_dl/extractor/youtube.py
View File

@ -1532,7 +1532,7 @@ class YoutubeSearchURLIE(InfoExtractor):
webpage = self._download_webpage(url, query) webpage = self._download_webpage(url, query)
result_code = self._search_regex( result_code = self._search_regex(
r'(?s)<ol class="item-section"(.*?)</ol>', webpage, 'result HTML') r'(?s)<ol[^>]+class="item-section"(.*?)</ol>', webpage, 'result HTML')
part_codes = re.findall( part_codes = re.findall(
r'(?s)<h3 class="yt-lockup-title">(.*?)</h3>', result_code) r'(?s)<h3 class="yt-lockup-title">(.*?)</h3>', result_code)

23
youtube_dl/options.py
View File

@ -195,6 +195,12 @@ def parseOpts(overrideArguments=None):
action='store_const', const='::', dest='source_address', action='store_const', const='::', dest='source_address',
help='Make all connections via IPv6 (experimental)', help='Make all connections via IPv6 (experimental)',
) )
network.add_option(
'--cn-verification-proxy',
dest='cn_verification_proxy', default=None, metavar='URL',
help='Use this proxy to verify the IP address for some Chinese sites. '
'The default proxy specified by --proxy (or none, if the options is not present) is used for the actual downloading. (experimental)'
)
selection = optparse.OptionGroup(parser, 'Video Selection') selection = optparse.OptionGroup(parser, 'Video Selection')
selection.add_option( selection.add_option(
@ -435,8 +441,12 @@ def parseOpts(overrideArguments=None):
downloader.add_option( downloader.add_option(
'--external-downloader', '--external-downloader',
dest='external_downloader', metavar='COMMAND', dest='external_downloader', metavar='COMMAND',
help='(experimental) Use the specified external downloader. ' help='Use the specified external downloader. '
'Currently supports %s' % ','.join(list_external_downloaders())) 'Currently supports %s' % ','.join(list_external_downloaders()))
downloader.add_option(
'--external-downloader-args',
dest='external_downloader_args', metavar='ARGS',
help='Give these arguments to the external downloader.')
workarounds = optparse.OptionGroup(parser, 'Workarounds') workarounds = optparse.OptionGroup(parser, 'Workarounds')
workarounds.add_option( workarounds.add_option(
@ -553,7 +563,7 @@ def parseOpts(overrideArguments=None):
action='store_true', dest='verbose', default=False, action='store_true', dest='verbose', default=False,
help='print various debugging information') help='print various debugging information')
verbosity.add_option( verbosity.add_option(
'--dump-intermediate-pages', '--dump-pages', '--dump-intermediate-pages',
action='store_true', dest='dump_intermediate_pages', default=False, action='store_true', dest='dump_intermediate_pages', default=False,
help='print downloaded pages to debug problems (very verbose)') help='print downloaded pages to debug problems (very verbose)')
verbosity.add_option( verbosity.add_option(
@ -729,6 +739,15 @@ def parseOpts(overrideArguments=None):
'--add-metadata', '--add-metadata',
action='store_true', dest='addmetadata', default=False, action='store_true', dest='addmetadata', default=False,
help='write metadata to the video file') help='write metadata to the video file')
postproc.add_option(
'--metadata-from-title',
metavar='FORMAT', dest='metafromtitle',
help='parse additional metadata like song title / artist from the video title. '
'The format syntax is the same as --output, '
'the parsed parameters replace existing values. '
'Additional templates: %(album), %(artist). '
'Example: --metadata-from-title "%(artist)s - %(title)s" matches a title like '
'"Coldplay - Paradise"')
postproc.add_option( postproc.add_option(
'--xattrs', '--xattrs',
action='store_true', dest='xattrs', default=False, action='store_true', dest='xattrs', default=False,

2
youtube_dl/postprocessor/__init__.py
View File

@ -16,6 +16,7 @@ from .ffmpeg import (
) )
from .xattrpp import XAttrMetadataPP from .xattrpp import XAttrMetadataPP
from .execafterdownload import ExecAfterDownloadPP from .execafterdownload import ExecAfterDownloadPP
from .metadatafromtitle import MetadataFromTitlePP
def get_postprocessor(key): def get_postprocessor(key):
@ -36,5 +37,6 @@ __all__ = [
'FFmpegPostProcessor', 'FFmpegPostProcessor',
'FFmpegSubtitlesConvertorPP', 'FFmpegSubtitlesConvertorPP',
'FFmpegVideoConvertorPP', 'FFmpegVideoConvertorPP',
'MetadataFromTitlePP',
'XAttrMetadataPP', 'XAttrMetadataPP',
] ]

6
youtube_dl/postprocessor/ffmpeg.py
View File

@ -549,7 +549,9 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
metadata['title'] = info['title'] metadata['title'] = info['title']
if info.get('upload_date') is not None: if info.get('upload_date') is not None:
metadata['date'] = info['upload_date'] metadata['date'] = info['upload_date']
if info.get('uploader') is not None: if info.get('artist') is not None:
metadata['artist'] = info['artist']
elif info.get('uploader') is not None:
metadata['artist'] = info['uploader'] metadata['artist'] = info['uploader']
elif info.get('uploader_id') is not None: elif info.get('uploader_id') is not None:
metadata['artist'] = info['uploader_id'] metadata['artist'] = info['uploader_id']
@ -558,6 +560,8 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
metadata['comment'] = info['description'] metadata['comment'] = info['description']
if info.get('webpage_url') is not None: if info.get('webpage_url') is not None:
metadata['purl'] = info['webpage_url'] metadata['purl'] = info['webpage_url']
if info.get('album') is not None:
metadata['album'] = info['album']
if not metadata: if not metadata:
self._downloader.to_screen('[ffmpeg] There isn\'t any metadata to add') self._downloader.to_screen('[ffmpeg] There isn\'t any metadata to add')

47
youtube_dl/postprocessor/metadatafromtitle.py Normal file
View File

@ -0,0 +1,47 @@
from __future__ import unicode_literals
import re
from .common import PostProcessor
from ..utils import PostProcessingError
class MetadataFromTitlePPError(PostProcessingError):
pass
class MetadataFromTitlePP(PostProcessor):
def __init__(self, downloader, titleformat):
super(MetadataFromTitlePP, self).__init__(downloader)
self._titleformat = titleformat
self._titleregex = self.format_to_regex(titleformat)
def format_to_regex(self, fmt):
"""
Converts a string like
'%(title)s - %(artist)s'
to a regex like
'(?P<title>.+)\ \-\ (?P<artist>.+)'
"""
lastpos = 0
regex = ""
# replace %(..)s with regex group and escape other string parts
for match in re.finditer(r'%\((\w+)\)s', fmt):
regex += re.escape(fmt[lastpos:match.start()])
regex += r'(?P<' + match.group(1) + '>.+)'
lastpos = match.end()
if lastpos < len(fmt):
regex += re.escape(fmt[lastpos:len(fmt)])
return regex
def run(self, info):
title = info['title']
match = re.match(self._titleregex, title)
if match is None:
raise MetadataFromTitlePPError('Could not interpret title of video as "%s"' % self._titleformat)
for attribute, value in match.groupdict().items():
value = match.group(attribute)
info[attribute] = value
self._downloader.to_screen('[fromtitle] parsed ' + attribute + ': ' + value)
return True, info

47
youtube_dl/utils.py
View File

@ -252,15 +252,12 @@ def sanitize_open(filename, open_mode):
raise raise
# In case of error, try to remove win32 forbidden chars # In case of error, try to remove win32 forbidden chars
alt_filename = os.path.join( alt_filename = sanitize_path(filename)
re.sub('[/<>:"\\|\\\\?\\*]', '#', path_part)
for path_part in os.path.split(filename)
)
if alt_filename == filename: if alt_filename == filename:
raise raise
else: else:
# An exception here should be caught in the caller # An exception here should be caught in the caller
stream = open(encodeFilename(filename), open_mode) stream = open(encodeFilename(alt_filename), open_mode)
return (stream, alt_filename) return (stream, alt_filename)
@ -305,11 +302,30 @@ def sanitize_filename(s, restricted=False, is_id=False):
result = result[2:] result = result[2:]
if result.startswith('-'): if result.startswith('-'):
result = '_' + result[len('-'):] result = '_' + result[len('-'):]
result = result.lstrip('.')
if not result: if not result:
result = '_' result = '_'
return result return result
def sanitize_path(s):
"""Sanitizes and normalizes path on Windows"""
if sys.platform != 'win32':
return s
drive, _ = os.path.splitdrive(s)
unc, _ = os.path.splitunc(s)
unc_or_drive = unc or drive
norm_path = os.path.normpath(remove_start(s, unc_or_drive)).split(os.path.sep)
if unc_or_drive:
norm_path.pop(0)
sanitized_path = [
path_part if path_part in ['.', '..'] else re.sub('(?:[/<>:"\\|\\\\?\\*]|\.$)', '#', path_part)
for path_part in norm_path]
if unc_or_drive:
sanitized_path.insert(0, unc_or_drive + os.path.sep)
return os.path.join(*sanitized_path)
def orderedSet(iterable): def orderedSet(iterable):
""" Remove all duplicates from the input iterable """ """ Remove all duplicates from the input iterable """
res = [] res = []
@ -1771,3 +1787,24 @@ def match_filter_func(filter_str):
video_title = info_dict.get('title', info_dict.get('id', 'video')) video_title = info_dict.get('title', info_dict.get('id', 'video'))
return '%s does not pass filter %s, skipping ..' % (video_title, filter_str) return '%s does not pass filter %s, skipping ..' % (video_title, filter_str)
return _match_func return _match_func
class PerRequestProxyHandler(compat_urllib_request.ProxyHandler):
def __init__(self, proxies=None):
# Set default handlers
for type in ('http', 'https'):
setattr(self, '%s_open' % type,
lambda r, proxy='__noproxy__', type=type, meth=self.proxy_open:
meth(r, proxy, type))
return compat_urllib_request.ProxyHandler.__init__(self, proxies)
def proxy_open(self, req, proxy, type):
req_proxy = req.headers.get('Ytdl-request-proxy')
if req_proxy is not None:
proxy = req_proxy
del req.headers['Ytdl-request-proxy']
if proxy == '__noproxy__':
return None # No Proxy
return compat_urllib_request.ProxyHandler.proxy_open(
self, req, proxy, type)

2
youtube_dl/version.py
View File

@ -1,3 +1,3 @@
from __future__ import unicode_literals from __future__ import unicode_literals
__version__ = '2015.02.28' __version__ = '2015.03.15'