WikipediR/0000755000176200001440000000000014604037454012150 5ustar liggesusersWikipediR/NAMESPACE0000644000176200001440000000217714601322242013363 0ustar liggesusers# Generated by roxygen2: do not edit by hand S3method(parse_response,actiontoken) S3method(parse_response,blink) S3method(parse_response,catpages) S3method(parse_response,createpage) S3method(parse_response,elink) S3method(parse_response,login) S3method(parse_response,pagecats) S3method(parse_response,pageinfo) S3method(parse_response,pcontent) S3method(parse_response,plink) S3method(parse_response,prelogintoken) S3method(parse_response,rchanges) S3method(parse_response,rcontent) S3method(parse_response,rdiff) S3method(parse_response,ucontribs) S3method(parse_response,uinfo) export(categories_in_page) export(create_pages) export(login) export(page_backlinks) export(page_content) export(page_external_links) export(page_info) export(page_links) export(pages_in_category) export(parse_response) export(query) export(random_page) export(recent_changes) export(revision_content) export(revision_diff) export(user_contributions) export(user_information) importFrom(httr,GET) importFrom(httr,content) importFrom(httr,stop_for_status) importFrom(httr,user_agent) importFrom(jsonlite,fromJSON) importFrom(magrittr,"%>%") importFrom(utils,URLencode) WikipediR/LICENSE0000644000176200001440000000005114601320617013142 0ustar liggesusersYEAR: 2014 COPYRIGHT HOLDER: Oliver KeyesWikipediR/README.md0000644000176200001440000000235514601342667013436 0ustar liggesusersWikipediR ========= An R API wrapper for MediaWiki, optimised for the Wikimedia Foundation MediaWiki instances, such as Wikipedia. __Author:__ Os Keyes
__License:__ [MIT](https://opensource.org/licenses/MIT)
__Status:__ Stable ![downloads](https://cranlogs.r-pkg.org/badges/grand-total/WikipediR) Description ====== _WikipediR_ is a wrapper around the MediaWiki API, optimised for the Wikimedia Foundation's production sites, such as Wikipedia. It is written in and for R, a statistical environment and associated programming language in heavy use by HCI researchers that, somehow, escaped having an API wrapper until now. Installation ====== For the most recent version, on CRAN: install.packages("WikipediR") For the development version: library(devtools) devtools::install_github("ironholds/WikipediR") Limitations ====== WikipediR currently lacks: * Quite a few possible API calls; * Direct authentication, through MediaWiki or OAuth. These issues will be solved in time; if there is a particular feature you want, open an issue here on github: I can't prioritise if I don't know what people are trying to do :). Dependencies ====== * R. Doy. * [httr](https://cran.r-project.org/package=httr) and its dependencies. WikipediR/man/0000755000176200001440000000000014601343046012715 5ustar liggesusersWikipediR/man/login.Rd0000644000176200001440000000106414601320617014314 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/login.R \name{login} \alias{login} \title{wikimedia api user login} \usage{ login(url, user, pw) } \arguments{ \item{url}{a URL body} \item{user}{a username of a registered user} \item{pw}{the password of said user} } \value{ TRUE } \description{ Login to a wikimedia instance to trigger api requests as a registered user. This function only allows the very basic login with username and password. Wikimedia setups that require more sophisticated login methods are not supported. } WikipediR/man/user_information.Rd0000644000176200001440000000732114604033777016604 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/user_info.R \name{user_information} \alias{user_information} \title{Retrieve user information} \usage{ user_information( language = NULL, project = NULL, domain = NULL, user_names, properties = c("blockinfo", "groups", "implicitgroups", "rights", "editcount", "registration", "emailable", "gender"), clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{user_names}{The username(s) of the users you want information on - this should be provided as a vector. There is a hard limit of 50 distinct users per query, set by MediaWiki's API; in the event that you go over this, a warning will be issued and the query will only be performed for the first 50 names in the vector.} \item{properties}{The user properties you're interested in. Applicable properties are "blockinfo" (details about the user's block, if they are currently blocked), "groups" (the user groups the user is a member of), "implicitgroups" (groups they are a member of through inheritance, as a result of membership in other groups), "rights" (what permissions their group membership grants them), "editcount" (how many non-deleted edits they have), "registration" (the date when they registered), "emailable" (whether they are contactable through Special:EmailUser) and "gender" (their provided gender).} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ Retrieves information about a user, or set of users, from the MediaWiki API, including registration date, gender and editcount. } \section{Warnings}{ There are a few caveats with the data provided by \code{user_information}, mostly stemming from historical inconsistencies and peculiarities in MediaWiki. \code{groups} and \code{implicitgroups} gives you the user's permissions and group membership on the project you are querying, not their membership on all projects - while you can find out if "Ironholds" is not a sysop on, say, enwiki, that doesn't mean they aren't a sysop elsewhere - there is no universal, API-accessible user groups listing. As an extension of the lack of centrality in Wikimedia's infrastructure, \code{registration} tells you the date their account was created on the wiki you are querying. If they initially registered on that wiki, this is accurate - if they registered on a different wiki, this instead reflects the date and time that they first visited the wiki you're querying while logged-in. For users registered before 2006, when registration logging was introduced, the \code{registration} value represents not when they first registered, but when their first edit was, since that was used as an estimator for existing accounts when the field was first populated. } \examples{ \dontrun{ #Retrieving information from a Wikimedia project user_info <- user_information("en", "wikipedia", user_names = "David Gerard", properties = "registration") #Non-Wikimedia projects user_info <- user_information(domain = "rationalwiki.org", user_names = "David Gerard", properties = "registration") } } \seealso{ \code{\link{user_contributions}} for retrieving recent contributions made by a particular user. } WikipediR/man/get_action_token.Rd0000644000176200001440000000054414601320617016522 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/create_content.R \name{get_action_token} \alias{get_action_token} \title{request token for api action as signed in user} \usage{ get_action_token(url) } \arguments{ \item{url}{a URL body} } \value{ a token string } \description{ helper function to request a user action token } WikipediR/man/page_backlinks.Rd0000644000176200001440000000420014604033777016147 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/metadata.R \name{page_backlinks} \alias{page_backlinks} \title{Retrieve a page's backlinks} \usage{ page_backlinks( language = NULL, project = NULL, domain = NULL, page, limit = 50, direction = "ascending", namespaces = NULL, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{page}{the title of the page you want the backlinks of.} \item{limit}{the number of backlinks to return. Set to 50 (the maximum) by default.} \item{direction}{the direction to order the backlinks in, by linking page ID: "ascending" or "descending". Set to "ascending" by default.} \item{namespaces}{The namespaces to filter to. By default, backlinks from any namespace are retrieved: alternately, a numeric vector of accepted namespaces (which are described \href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be provided, and only backlinks from pages within those namespaces will be returned.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ page_backlinks, when provided with a page title, retrieves backlinks to that page. Output can be filtered to specific namespaces. } \section{Warnings}{ as with \code{\link{pages_in_category}}, if the page you are linking to does not exist, an empty list will be returned, without any indication of an error. } \examples{ \dontrun{ #Backlink all_bls <- page_backlinks("en","wikipedia", page = "Aaron Halfaker") #Namespace-specific backlinks mainspace_bls <- page_backlinks("en","wikipedia", page = "Aaron Halfaker", namespaces = 0) } } WikipediR/man/get_prelogin_token.Rd0000644000176200001440000000061114601320617017057 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/login.R \name{get_prelogin_token} \alias{get_prelogin_token} \title{request token to start client login} \usage{ get_prelogin_token(url, user) } \arguments{ \item{url}{a URL body} \item{user}{a username of a registered user} } \value{ a token string } \description{ helper function to request a user login token } WikipediR/man/user_contributions.Rd0000644000176200001440000000576714604033777017175 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/user_info.R \name{user_contributions} \alias{user_contributions} \title{Retrieve user contributions} \usage{ user_contributions( language = NULL, project = NULL, domain = NULL, username, properties = c("ids", "title", "timestamp", "comment", "parsedcomment", "size", "sizediff", "flags", "tags"), mainspace = FALSE, limit = 50, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{username}{The username of the user whose contributions you want to retrieve. Due to limitations at the API end, you can only retrieve edits for one user at a time.} \item{properties}{The metadata you want associated with each edit. Potential metadata includes "ids" (the revision ID of the revision, which can be passed into \code{\link{revision_content}}), "title" (the name of the page that was edited), "timestamp", "comment" (the edit summary associated with the revision), "parsedcomment" (the same, but parsed, generating HTML from any wikitext in that comment), "size" (the size, in uncompressed bytes, of the edit), "sizediff" (the size delta between this edit, and the last edit to the page), "flags" (whether the revision was 'minor' or not), and "tags" (any tags associated with the revision).} \item{mainspace}{A boolean flag; FALSE retrieves all of the most recent contributions, while TRUE limits the retrieved contributions to those in the 'mainspace' - in other words, edits to actual articles. Set to FALSE by default} \item{limit}{The number of edits to be retrieved. 50 is the maximum for logged-out API users, and putting in more than 50 will generate a warning.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ Retrieves metadata associated with the most recent contributions by a specified user. } \examples{ \dontrun{ #Retrieve the timestamps of a user's recent contributions to the English-language Wikipedia contribs <- user_contributions("en", "wikipedia", username = "Ironholds", properties = "timestamp") #Retrieve the timestamps of a user's recent contributions to a non-Wikimedia wiki. rw_contribs <- user_contributions(domain = "rationalwiki.org", username = "David Gerard", properties = "ids", limit = 1) } } \seealso{ \code{\link{user_information}} for information about a specific user (or group of users), and \code{recent_changes} for non-user-specific recent actions. } WikipediR/man/recent_changes.Rd0000644000176200001440000000576014601321055016160 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/recent_changes.R \name{recent_changes} \alias{recent_changes} \title{Retrieves entries from the RecentChanges feed} \usage{ recent_changes( language = NULL, project = NULL, domain = NULL, properties = c("user", "userid", "comment", "parsedcomment", "flags", "timestamp", "title", "ids", "sizes", "redirect", "loginfo", "tags", "sha1"), type = c("edit", "external", "new", "log"), tag = NULL, dir = "newer", limit = 50, top = FALSE, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{properties}{Properties you're trying to retrieve about each entry, Options include "user" (the username of the person responsible for that entry), "userid" (the userID of said person), "comment" (the edit summary associated with the entry), "parsedcomment" (the same, but parsed, generating HTML from any wikitext in that comment), "flags" (whether the revision was 'minor' or not), "timestamp", "title" (the name of the page the entry affected), "ids" (the page id, along with the old and new revision IDs when applicable) "sizes" (the size, in uncompressed bytes, of the entry, and, in the case of revisions, the size of the edit it displaced), "tags" (any tags associated with the revision) and "loginfo" (applicable only to log entries, and consisting of log ID numbers, log types and actions, and so on) and "sha1" (the SHA-1 hash of the revision text).} \item{type}{The type of entry you want to retrieve; can be any permutation of "edit" (edits to existing pages), "external" (external actions that impact on the project - primarily wikidata changes), "new" (the creation of new pages) and "log" (log entries). By default, all of these entry types are included.} \item{tag}{Only return items with particular "tags", such as "mobile edit". NULL by default.} \item{dir}{Should it go from newest to oldest ("newer"), or oldest to newest ("older")? By default, set to "newer".} \item{limit}{The number of entries you'd like to return. By default, set to 50, which is also the maximum number per-request for logged-out users.} \item{top}{Should the request only return "top" entries - in other words, the most recent entry on a page? Set to FALSE by default.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ wiki_recentchanges retrieves a stream of entries from Special:RecentChanges, with a variety of associated metadata and filtering (of both entries *and* that metadata. } WikipediR/man/create_page.Rd0000644000176200001440000000136414601320617015446 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/create_content.R \name{create_page} \alias{create_page} \title{wikimedia api page creation (single pages) helper function to do the actual api requests for page and category-page creation} \usage{ create_page(url, p_title, p_text, category, token) } \arguments{ \item{url}{a URL body} \item{p_title}{page title string of new page} \item{p_text}{page content string of new page} \item{category}{switch to decide, if the page should be created as category-page} \item{token}{action token to perform the request} } \value{ TRUE } \description{ wikimedia api page creation (single pages) helper function to do the actual api requests for page and category-page creation } WikipediR/man/query.Rd0000644000176200001440000000126614601320617014355 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/query.R \name{query} \alias{query} \title{base query function} \usage{ query(url, out_class, clean_response = FALSE, query_param = list(), ...) } \arguments{ \item{url}{a URL body} \item{out_class}{the class to set on the output object; used within WikidataR to indicate what response-cleaning method should be applied.} \item{clean_response}{whether to clean the response, using the method assigned by out_class, or not.} \item{query_param}{query parameters} \item{...}{further arguments to httr's GET.} } \description{ not designed to be used by anyone except a third-party reuser package, such as WikidataR } WikipediR/man/revision_diff.Rd0000644000176200001440000000650514604033777016052 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/content.R \name{revision_diff} \alias{revision_diff} \title{Generates a "diff" between a pair of revisions} \usage{ revision_diff( language = NULL, project = NULL, domain = NULL, revisions, properties = c("ids", "flags", "timestamp", "user", "userid", "size", "sha1", "contentmodel", "comment", "parsedcomment", "tags", "flagged"), direction, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{revisions}{The revision IDs of each "start" revision.} \item{properties}{Properties you're trying to retrieve about that revision, should you want to; options include "ids" (the revision ID of the revision...which is pointless), "flags" (whether the revision was 'minor' or not), "timestamp","user" (the username of the person who made that revision), "userid" (the userID of the person who made the revision), "size" (the size, in uncompressed bytes, of the revision), "sha1" (the SHA-1 hash of the revision text), "contentmodel" (the content model of the page, usually "wikitext"), "comment" (the revision summary associated with the revision), "parsedcomment" (the same, but parsed, generating HTML from any wikitext in that comment), "tags" (any tags associated with the revision) and "flagged" (the revision's status under Flagged Revisions).} \item{direction}{The direction you want the diff to go in from the revisionID you have provided. Options are "prev" (compare to the previous revision on that page), "next" (compare to the next revision on that page) and "cur" (compare to the current, extant version of the page).} \item{clean_response}{whether to do some basic sanitising of the resulting data structure.} \item{...}{further arguments to pass to httr's GET.} } \description{ revision_diff generates a diff between two revisions in a MediaWiki page. This is provided as an XML-parsable blob inside the returned JSON object. } \section{Warnings}{ MediaWiki's API is deliberately designed to restrict users' ability to make computing-intense requests - such as diff computation. As a result, the API only allows requests for one uncached diff in each request. If you ask for multiple diffs, some uncached and some cached, you will be provided with the cached diffs, one of the uncached diffs, and a warning. If you're going to be asking for a lot of diffs, some of which may not be cached, it may be more sensible to retrieve the revisions themselves using \code{\link{revision_content}} and compute the diffs yourself. } \examples{ \dontrun{ #Wikimedia diff wp_diff <- revision_diff("en","wikipedia", revisions = 552373187, direction = "next") #Non-Wikimedia diff rw_diff <- revision_diff(domain = "rationalwiki.org", revisions = 88616, direction = "next") } } \seealso{ \code{\link{page_content}} for retrieving the current content of a specific page, and \code{\link{revision_content}} for retrieving the text of specific revisions. } WikipediR/man/categories_in_page.Rd0000644000176200001440000000433614604033776017032 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/categories.R \name{categories_in_page} \alias{categories_in_page} \title{Retrieves categories associated with a page.} \usage{ categories_in_page( language = NULL, project = NULL, domain = NULL, pages, properties = c("sortkey", "timestamp", "hidden"), limit = 50, show_hidden = FALSE, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{pages}{A vector of page titles, with or without spaces, that you want to retrieve categories for.} \item{properties}{The properties you want to retrieve about the categories. Options are "sortkey" (the key that sorts the way the page is stored in each category), "timestamp" (when the category was added to that page) and "hidden" (tags those categories in the returned list that are 'hidden' from readers).} \item{limit}{The maximum number of categories you want to retrieve for each page. Set to 50 by default.} \item{show_hidden}{Whether or not to include 'hidden' categories in the categories that are retrieved - these are usually associated with the maintenance of Wikipedia and its internal processes. Set to FALSE by default.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ Retrieves categories associated with a page (or list of pages) on a MediaWiki instance } \examples{ \dontrun{ #Retrieve the categories for the "New Age" article on en.wiki cats <- categories_in_page("en", "wikipedia", pages = "New Age") #Retrieve the categories for the "New Age" article on rationalwiki. rw_cats <- categories_in_page(domain = "rationalwiki.org", pages = "New Age") } } \seealso{ \code{\link{pages_in_category}} for pages in a specified category. } WikipediR/man/random_page.Rd0000644000176200001440000000400114604033776015464 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/content.R \name{random_page} \alias{random_page} \title{Retrieve the page content of a random MediaWiki page} \usage{ random_page( language = NULL, project = NULL, domain = NULL, namespaces = NULL, as_wikitext = FALSE, limit = 1, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{namespaces}{The namespaces to consider pages from. By default, pages from any namespace are considered; alternately, a numeric vector of accepted namespaces (which are described \href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be provided, and only pages within those namespaces will be considered.} \item{as_wikitext}{whether to retrieve the wikimarkup (TRUE) or the HTML (FALSE). Set to FALSE by default.} \item{limit}{the number of pages to return. 1 by default.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ wiki_page retrieves the DOM of a particular MediaWiki page, as a HTML blob inside a JSON object. } \examples{ \dontrun{ #A page from Wikipedia wp_content <- random_page("en","wikipedia") #A page from the mainspace on Wikipedia wp_article_content <- random_page("en","wikipedia", namespaces = 0) } } \seealso{ \code{\link{page_content}} for retrieving the content of a specific page, \code{\link{revision_diff}} for retrieving 'diffs' between revisions, \code{\link{revision_content}} for retrieving the text of specified revisions. } WikipediR/man/page_info.Rd0000644000176200001440000000327314604033777015152 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/metadata.R \name{page_info} \alias{page_info} \title{Retrieve information about a particular page} \usage{ page_info( language = NULL, project = NULL, domain = NULL, page, properties = c("protection", "talkid", "url", "displaytitle"), clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{page}{the title of the page you want the metadata of.} \item{properties}{the properties you'd like to retrieve. Some properties (the pageID, namespace, title, language, length and most recent revision ID, for example) are retrieved by default, whatever is passed to \code{properties}: properties that can be explicitly retrieved include the page's protection level ("protection"), the ID of the associated talk page, if applicable ("talkid"), the full, canonical URL ("url"), and the displayed page title ("displaytitle").} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ page_info, when provided with a page title, retrieves metadata about that page. } \examples{ \dontrun{ #Metadata page_metadata <- page_info("en","wikipedia", page = "Aaron Halfaker") } } WikipediR/man/page_external_links.Rd0000644000176200001440000000316214604033777017236 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/metadata.R \name{page_external_links} \alias{page_external_links} \title{Retrieve a page's links} \usage{ page_external_links( language = NULL, project = NULL, domain = NULL, page, protocol = NULL, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{page}{the title of the page you want the links of.} \item{protocol}{limit links to those with certain link protocols. Options are listed in Special:ApiSandbox's \href{https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extlinks}{elprotocol field}.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ page_external_links, when provided with a page title, retrieves external wikilinks from the current revision of that page. } \examples{ \dontrun{ #Links external_links <- page_external_links("en","wikipedia", page = "Aaron Halfaker") #Protocol-specific links external_http_links <- page_external_links("en","wikipedia", page = "Aaron Halfaker", protocol = "http") } } WikipediR/man/parse_response.Rd0000644000176200001440000000053414601343046016236 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/parse.R \name{parse_response} \alias{parse_response} \title{parse_response: Parse WikipediR responses internally} \usage{ parse_response(x) } \arguments{ \item{x}{result from a WikipediR query} } \description{ Response parser } \details{ Should not be externally used } WikipediR/man/page_links.Rd0000644000176200001440000000364114604033777015336 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/metadata.R \name{page_links} \alias{page_links} \title{Retrieve a page's links} \usage{ page_links( language = NULL, project = NULL, domain = NULL, page, limit = 50, direction = "ascending", namespaces = NULL, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{page}{the title of the page you want the links of.} \item{limit}{the number of links to retrieve. 50 by default; a maximum of 500 is set server-side.} \item{direction}{the direction to order the links in, by destination page ID: "ascending" or "descending". Set to "ascending" by default.} \item{namespaces}{The namespaces to filter to. By default, links to any namespace are retrieved: alternately, a numeric vector of accepted namespaces (which are described \href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be provided, and only backlinks from pages within those namespaces will be returned.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ page_links, when provided with a page title, retrieves internal wikilinks from the current revision of that page. } \examples{ \dontrun{ #Links links <- page_links("en","wikipedia", page = "Aaron Halfaker") #Namespace-specific links mainspace_links <- page_links("en","wikipedia", page = "Aaron Halfaker", namespaces = 0) } } WikipediR/man/create_pages.Rd0000644000176200001440000000121714601320617015626 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/create_content.R \name{create_pages} \alias{create_pages} \title{wikimedia api page creation Create pages or category-pages on a wikimedia instance.} \usage{ create_pages(url, p_title, p_text, category = FALSE) } \arguments{ \item{url}{a URL body} \item{p_title}{vector with page title strings of new pages} \item{p_text}{vector with page content strings of new pages} \item{category}{switch to decide, if the pages should be created as category-pages} } \value{ TRUE } \description{ wikimedia api page creation Create pages or category-pages on a wikimedia instance. } WikipediR/man/WikipediR.Rd0000644000176200001440000000111414601321055015064 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/WikipediR.R \docType{package} \name{WikipediR} \alias{WikipediR} \alias{WikipediR-package} \title{A client library for MediaWiki's API} \description{ This package provides functions for accessing the MediaWiki API, either for Wikimedia projects or any other MediaWiki instance. For more information, see the \href{https://CRAN.R-project.org/package=WikipediR/vignettes/WikipediR.html}{vignette}. } \seealso{ The \href{https://CRAN.R-project.org/package=WikipediR/vignettes/WikipediR.html}{package vignette}. } WikipediR/man/revision_content.Rd0000644000176200001440000000471214604033777016612 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/content.R \name{revision_content} \alias{revision_content} \title{Retrieves MediaWiki revisions} \usage{ revision_content( language = NULL, project = NULL, domain = NULL, revisions, properties = c("content", "ids", "flags", "timestamp", "user", "userid", "size", "sha1", "contentmodel", "comment", "parsedcomment", "tags"), clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{revisions}{The revision IDs of each desired revision.} \item{properties}{Properties you're trying to retrieve about that revision, should you want to; options include "ids" (the revision ID of the revision...which is pointless), "flags" (whether the revision was 'minor' or not), "timestamp" (the timestamp of the revision), "user" (the username of the person who made that revision), "userid" (the userID of the person who made the revision), "size" (the size, in uncompressed bytes, of the revision), "sha1" (the SHA-1 hash of the revision text), "contentmodel" (the content model of the page, usually "wikitext"), "comment" (the revision summary associated with the revision), "parsedcomment" (the same, but parsed, generating HTML from any wikitext in that comment), "tags" (any tags associated with the revision) and "flagged" (the revision's status under Flagged Revisions).} \item{clean_response}{whether to do some basic sanitising of the resulting data structure.} \item{...}{further arguments to pass to httr's GET.} } \description{ Retrieves the content of a provided list of revisions from whichever MediaWiki instance you're querying. Returns as wikimarkup. } \examples{ \dontrun{ #Revision content from a Wikimedia project wp_content <- revision_content("en","wikipedia", revisions = 552373187) #Revision content from a non-Wikimedia project rw_content <- revision_content(domain = "rationalwiki.org", revisions = 88616) } } \seealso{ \code{\link{revision_diff}} for diffs between revisions, and \code{\link{page_content}} for the content a specific page currently has. } WikipediR/man/pages_in_category.Rd0000644000176200001440000000513114604033776016677 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/categories.R \name{pages_in_category} \alias{pages_in_category} \title{Retrieves a list of category members.} \usage{ pages_in_category( language = NULL, project = NULL, domain = NULL, categories, properties = c("title", "ids", "sortkey", "sortkeyprefix", "type", "timestamp"), type = c("page", "subcat", "file"), clean_response = FALSE, limit = 50, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{categories}{The names of the categories you want to gather information for.} \item{properties}{The properties you want to gather for each member of the category. Options are "title" (the name of the member, including namespace), "id" (the unique numeric identifier of the member), "sortkey" (the hexadecimal key used to sort that member within the category), "sortkeyprefix" (the human-readable sort key), "type" (whether the member is a page, a subcategory or a file) and "timestamp" (when the member was added to the category)} \item{type}{The type of member you're interested in returning; options are any permutation of "page" (pages), "subcat" (subcategories) and "file" (files).} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{limit}{The maximum number of members to retrieve for each category. Set to 50 by default.} \item{...}{further arguments to pass to httr's GET().} } \description{ wiki_catpages retrieves a list of pages, subcategories, files or all of the above in a specified category (or series of specified categories) } \section{warnings}{ Because of the way MediaWiki stores this data, both "the category you asked for doesn't exist" and "the category you asked for exists, but has no members" return in the same way. } \examples{ \dontrun{ #Retrieve the pages in the "New Age" category on en.wiki cats <- pages_in_category("en", "wikipedia", categories = "New Age") #Retrieve the pages in the "New Age" category on rationalwiki. rw_cats <- pages_in_category(domain = "rationalwiki.org", categories = "New Age") } } \seealso{ \code{\link{categories_in_page}} for finding categories that a specified page is a member of. } WikipediR/man/page_content.Rd0000644000176200001440000000355714604033777015676 0ustar liggesusers% Generated by roxygen2: do not edit by hand % Please edit documentation in R/content.R \name{page_content} \alias{page_content} \title{Retrieves MediaWiki page content} \usage{ page_content( language = NULL, project = NULL, domain = NULL, page_name, page_id = NULL, as_wikitext = FALSE, clean_response = FALSE, ... ) } \arguments{ \item{language}{The language code of the project you wish to query, if appropriate.} \item{project}{The project you wish to query ("wikiquote"), if appropriate. Should be provided in conjunction with \code{language}.} \item{domain}{as an alternative to a \code{language} and \code{project} combination, you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing for the querying of non-Wikimedia MediaWiki instances.} \item{page_name}{The title of the page you want to retrieve} \item{page_id}{the pageID of the page you want to retrieve. Set to NULL by default, and an alternative to page_name; if both are provided, page_id will be used.} \item{as_wikitext}{whether to retrieve the wikimarkup (TRUE) or the HTML (FALSE). Set to FALSE by default.} \item{clean_response}{whether to do some basic sanitising of the resulting data structure. Set to FALSE by default.} \item{...}{further arguments to pass to httr's GET.} } \description{ wiki_page retrieves the DOM of a particular MediaWiki page, as a HTML blob inside a JSON object. } \examples{ \dontrun{ #Content from a Wikimedia project wp_content <- page_content("en","wikipedia", page_name = "Aaron Halfaker") #Content by ID wp_content <- page_content("en", "wikipedia", page_id = 12) #Content from a non-Wikimedia project rw_content <- page_content(domain = "rationalwiki.org", page_name = "New Age") } } \seealso{ \code{\link{revision_diff}} for retrieving 'diffs' between revisions, \code{\link{revision_content}} for retrieving the text of specified revisions. } WikipediR/DESCRIPTION0000644000176200001440000000153514604037454013662 0ustar liggesusersPackage: WikipediR Type: Package Title: A MediaWiki API Wrapper Version: 1.7.1 Date: 2024-04-05 Author: Os Keyes [aut, cre], Brock Tilbert [ctb], Clemens Schmid [aut] Maintainer: Os Keyes Description: A wrapper for the MediaWiki API, aimed particularly at the Wikimedia 'production' wikis, such as Wikipedia. It can be used to retrieve page text, information about users or the history of pages, and elements of the category tree. License: MIT + file LICENSE Imports: httr, jsonlite, magrittr Suggests: testthat, knitr, WikidataR, pageviews BugReports: https://github.com/Ironholds/WikipediR/issues URL: https://github.com/Ironholds/WikipediR/ VignetteBuilder: knitr RoxygenNote: 7.3.1 Encoding: UTF-8 NeedsCompilation: no Packaged: 2024-04-05 17:43:38 UTC; ironholds Repository: CRAN Date/Publication: 2024-04-05 18:13:00 UTC WikipediR/build/0000755000176200001440000000000014604034112013233 5ustar liggesusersWikipediR/build/vignette.rds0000644000176200001440000000030214604034112015565 0ustar liggesusersb```b`add`b2 1# ' ,HM MAKI!teɂ!@„$ϚZ% 5/$~hZ8S+`zP԰Aհe ,s\ܠL t7`~΢r=xAmݣ9JI,IK+iWikipediR/tests/0000755000176200001440000000000014601320617013303 5ustar liggesusersWikipediR/tests/testthat/0000755000176200001440000000000014604037454015152 5ustar liggesusersWikipediR/tests/testthat/test_category_retrieval.R0000644000176200001440000000205114601320737022220 0ustar liggesuserscontext("Category retrieval") test_that("page categories can be retrieved through categories_in_page", { skip_on_cran() expect_true({categories_in_page("en","wikipedia", page = "Barack Obama");TRUE}) }) test_that("Hidden page categories can be retrieved through categories_in_page", { skip_on_cran() expect_true({categories_in_page("en","wikipedia", page = "Barack Obama", show_hidden=T);TRUE}) }) test_that("Category members can be retrieved through categories_in_page", { skip_on_cran() expect_true({pages_in_category("en","wikipedia", categories = "1920 births");TRUE}) }) test_that("Category members can be retrieved through categories_in_page", { skip_on_cran() expect_true({pages_in_category("en","wikipedia", categories = "1920s births", type = "subcat");TRUE}) }) test_that("page categories can be retrieved through categories_in_page with non-ASCII query", { skip_on_cran() # \u30dd\u30b1\u30e2\u30f3 is "Pokemon" in Japanese letters expect_true({categories_in_page("en","wikipedia", page = "\u30dd\u30b1\u30e2\u30f3");TRUE}) })WikipediR/tests/testthat/test_recent_changes.R0000644000176200001440000000120314601321044021264 0ustar liggesuserscontext("Recent changes") test_that("Recentchanges feed entries can be retrieved through recent_changes", { skip_on_cran() expect_true({recent_changes("en","wikipedia", limit=1);TRUE}) }) test_that("recent_changes respects tags", { skip_on_cran() expect_true({recent_changes("en","wikipedia", limit=1, tag = "mobile edit");TRUE}) }) test_that("recent_changes respects types", { skip_on_cran() expect_true({recent_changes("en","wikipedia", limit=1, type = "new");TRUE}) }) test_that("recent_changes respects directional changes", { skip_on_cran() expect_true({recent_changes("en","wikipedia", limit=1, top = TRUE);TRUE}) }) WikipediR/tests/testthat/test_content_retrieval.R0000644000176200001440000000231514601320736022057 0ustar liggesuserscontext("Content retrieval") test_that("Wikitext content can be retrieved through page_content", { skip_on_cran() expect_true({page_content("en","wikipedia", page_name = "Barack Obama", as_wikitext=TRUE);TRUE}) expect_true({page_content("en","wikipedia", page_id = "534366", as_wikitext=TRUE);TRUE}) }) test_that("HTML content can be retrieved through page_content", { skip_on_cran() expect_true({page_content("en","wikipedia", page_name = "Barack Obama", as_wikitext=FALSE);TRUE}) expect_true({page_content("en","wikipedia", page_id = "534366", as_wikitext=FALSE);TRUE}) }) test_that("HTML content can be retrieved through revision_content", { skip_on_cran() expect_true({revision_content("en","wikipedia", revisions = "102342934");TRUE}) }) test_that("Diffs can be retrieved through revision_content", { skip_on_cran() expect_true({revision_diff("en","wikipedia", revisions = "129122231", direction = "next");TRUE}) }) test_that("Wikitext content can be retrieved through page_content with non-ASCII query", { skip_on_cran() # \u30dd\u30b1\u30e2\u30f3 is "Pokemon" in Japanese letters expect_true({page_content("en","wikipedia", page_name = "\u30dd\u30b1\u30e2\u30f3", as_wikitext=TRUE);TRUE}) })WikipediR/tests/testthat/test_metadata_retrieval.R0000644000176200001440000000270214601321032022153 0ustar liggesuserscontext("Metadata") test_that("backlinks can be retrieved through page_backlinks", { skip_on_cran() expect_true({page_backlinks("en","wikipedia", page = "Aaron Halfaker");TRUE}) }) test_that("backlinks from a specific namespace can be retrieved through page_backlinks", { skip_on_cran() expect_true({page_backlinks("en","wikipedia", page = "Aaron Halfaker", namespaces = 0);TRUE}) }) test_that("links can be retrieved through page_links", { skip_on_cran() expect_true({page_links("en","wikipedia", page = "Aaron Halfaker");TRUE}) }) test_that("links from a specific namespace can be retrieved through page_links", { skip_on_cran() expect_true({page_links("en","wikipedia", page = "Aaron Halfaker", namespaces = 0);TRUE}) }) test_that("external links can be retrieved through page_links", { skip_on_cran() expect_true({page_external_links("en","wikipedia", page = "Aaron Halfaker");TRUE}) }) test_that("external links with a particular protocol can be retrieved through page_links", { skip_on_cran() expect_true({page_external_links("en","wikipedia", page = "Aaron Halfaker", protocol = "http");TRUE}) }) test_that("page info can be retrieved through page_info", { skip_on_cran() expect_true({page_info("en","wikipedia", page = "Aaron Halfaker");TRUE}) }) test_that("page info with specified params can be retrieved through page_info", { expect_true({page_info("en","wikipedia", page = "Aaron Halfaker", properties = "talkid");TRUE}) }) WikipediR/tests/testthat.R0000644000176200001440000000007514601320617015270 0ustar liggesuserslibrary(WikipediR) library(testthat) test_check("WikipediR") WikipediR/vignettes/0000755000176200001440000000000014604034112014144 5ustar liggesusersWikipediR/vignettes/WikipediR.Rmd0000644000176200001440000000363214601320617016510 0ustar liggesusers #WikipediR: A MediaWiki API client library Many websites run on versions of MediaWiki, most prominently Wikipedia and its sister sites. WikipediR is an API client library that allows you to conveniently make requests for content and associated metadata against MediaWiki instances. ## Retrieving content "content" can mean a lot of different things - but mostly, we mean the text of an article, either its current version or any previous versions. Current versions can be retrieved using page\_content, which provides both HTML and wikitext as possible output formats. Older, individual revisions can be retrieved with revision\_content. These functions also return a range of possible metadata about the revisions or articles in question. Diffs between revisions can be generated using revision\_diff, while individual ''elements'' of a page's content - particularly links - can be extracted using page\_links, page\_backlinks, and page\_external\_links. And if the interest is in changes to content, rather than content itself, recent\_changes can be used to grab a slice of a project's Special:RecentChanges feed. ## Retrieving metadata Page-related information can be accessed using page\_info, while categories that a page possesses can be retrieved with categories\_in\_page - the inverse of this operation (what pages are in a particular category?) uses pages\_in\_category. User-related info can be accessed with user\_information, while user\_contributions allows access to recent contributions by a particular user: this can be conveniently linked up with revision\_content, mentioned above, to retrieve the content of the last N edits by a particular editor, or metadata about those edits.WikipediR/NEWS0000644000176200001440000000636214601320617012647 0ustar liggesusersVersion 1.5.0 ------------------------------------------------------------------------------ BUG FIXES * https is now the default for requests (Thanks to Hiroaki Yutani) * UTF-8 page names and similar are properly handled (Thanks to Hiroaki Yutani) Version 1.4.0 ------------------------------------------------------------------------------ NEW FEATURES * `random_page` now allows you to return more than one page. * `page_backlinks` does as well; thanks to Brock Tibert for the code. DOCUMENTATION * Vignette tweaks made * Package-level documentation improved. DEVELOPMENT * "Depends" switched out for "Imports" to avoid annoying loading messages. Version 1.3.0 ------------------------------------------------------------------------------ BUG FIXES * Various miscellaneous internal improvements. * `limit` introduced as an argument to pages_in_category - thanks to Ben Marwick for finding the bug. * `limit` introduced as an argument to page_links - thanks to Hui Li of the Universität Heidelberg for the initial report. Version 1.2.0 ------------------------------------------------------------------------------ NEW FEATURES *Random pages can now be retrieved with random_page *pageID based querying is available *Custom user DEVELOPMENT *Doc tweaks and some simplification of argument parsing around page_content and random_page *Additional unit tests *query() marked as exportable to allow simple third-party plugins into WikipediR Version 1.0.1 ------------------------------------------------------------------------------ * Roxygenised documentation, some R CHECK errors fixed. * Test suite added for error handlers. Version 1.0.0 ------------------------------------------------------------------------------ * First release version * Introduction of wiki_usercontribs, which retrieves recent contributions for a specified username * Bugfixes to error handling routines * Bugfix to problems around wiki_con, where species or commons users were unable to take advantage of CurlOpts. Version 0.7.0 ------------------------------------------------------------------------------ * wiki_page now returns the DOM of the relevant page, not the wikimarkup, in line with a suggestion from Scott Chamberlain. This should be parsable by the HTML tree parser in the XML package. Version 0.6.3 ------------------------------------------------------------------------------ * Standardisation and cleanup of error handlers, and the introduction of a single central LimitHandler. Version 0.6.2 ------------------------------------------------------------------------------ * Bug fix from Scott Chamberlain around wiki_page, switching the content type over to x-wikitext. * Introduction of w_timeout as a field in the connector object; will be extended to other CURL options in time. Version 0.6.1 ------------------------------------------------------------------------------ * Support for Commons and Wikispecies * Addition of wiki_recentchanges(), which allows a useR to query the recentchanges feed. Version 0.5.1 ------------------------------------------------------------------------------ * Format for wiki_revision changed from the (unacceptable-to-API) text/css to x/wikitext. * Error handling improved to pass the full error explanation through to the user, in the case of API errors.WikipediR/R/0000755000176200001440000000000014601320617012342 5ustar liggesusersWikipediR/R/categories.R0000644000176200001440000001447314604033612014622 0ustar liggesusers#'@title Retrieves categories associated with a page. #' #'@description #'Retrieves categories associated with a page (or list of pages) on a MediaWiki instance #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param pages A vector of page titles, with or without spaces, that you want to retrieve #'categories for. #' #'@param properties The properties you want to retrieve about the categories. #'Options are "sortkey" (the key that sorts the way the page is stored in each category), #'"timestamp" (when the category was added to that page) and "hidden" (tags those categories #'in the returned list that are 'hidden' from readers). #' #'@param limit The maximum number of categories you want to retrieve for each page. Set #'to 50 by default. #' #'@param show_hidden Whether or not to include 'hidden' categories in the categories #'that are retrieved - these are usually associated with the maintenance of Wikipedia #'and its internal processes. Set to FALSE by default. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@seealso \code{\link{pages_in_category}} for pages in a specified category. #' #'@examples #'\dontrun{ #'#Retrieve the categories for the "New Age" article on en.wiki #'cats <- categories_in_page("en", "wikipedia", pages = "New Age") #' #'#Retrieve the categories for the "New Age" article on rationalwiki. #'rw_cats <- categories_in_page(domain = "rationalwiki.org", pages = "New Age") #'} #'@export categories_in_page <- function(language = NULL, project = NULL, domain = NULL, pages, properties = c("sortkey","timestamp","hidden"), limit = 50, show_hidden = FALSE, clean_response = FALSE, ...){ #Check, construct URL properties <- match.arg(properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") pages <- handle_limits(pages, 50) if(show_hidden){ show_hidden <- "hidden" } else { show_hidden <- "!hidden" } url <- url_gen(language, project, domain) query_param <- list( action = "query", prop = "categories", clprop = properties, clshow = show_hidden, cllimit= limit, titles = pages ) #Retrieve, check, return content <- query(url, "pagecats", clean_response, query_param = query_param, ...) page_names <- names(unlist(content)) missing_pages <- sum(grepl(x = page_names, pattern = "missing")) if(missing_pages){ warning("This request contained ",missing_pages," invalid page title(s)", call. = FALSE) } return(content) } #'@title #'Retrieves a list of category members. #' #'@description #'wiki_catpages retrieves a list of pages, subcategories, files or all of the above #'in a specified category (or series of specified categories) #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param categories The names of the categories you want to gather information for. #' #'@param properties The properties you want to gather for each member of the category. #'Options are "title" (the name of the member, including namespace), #'"id" (the unique numeric identifier of the member), "sortkey" #'(the hexadecimal key used to sort that member within the category), #'"sortkeyprefix" (the human-readable sort key), "type" #'(whether the member is a page, a subcategory or a file) and #'"timestamp" (when the member was added to the category) #' #'@param type The type of member you're interested in returning; #'options are any permutation of "page" (pages), "subcat" (subcategories) and "file" (files). #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param limit The maximum number of members to retrieve for each category. Set #'to 50 by default. #' #'@param ... further arguments to pass to httr's GET(). #' #'@section warnings: #'Because of the way MediaWiki stores this data, both "the category you asked for doesn't exist" #'and "the category you asked for exists, but has no members" return in the same way. #' #'@seealso \code{\link{categories_in_page}} for finding categories that a specified page is a member of. #' #'@examples #'\dontrun{ #'#Retrieve the pages in the "New Age" category on en.wiki #'cats <- pages_in_category("en", "wikipedia", categories = "New Age") #' #'#Retrieve the pages in the "New Age" category on rationalwiki. #'rw_cats <- pages_in_category(domain = "rationalwiki.org", categories = "New Age") #'} #'@export pages_in_category <- function(language = NULL, project = NULL, domain = NULL, categories, properties = c("title","ids","sortkey","sortkeyprefix","type","timestamp"), type = c("page","subcat","file"), clean_response = FALSE, limit = 50, ...){ #Format and check categories <- gsub(x = categories, pattern = "^", replacement = "Category:") categories <- handle_limits(categories, 50) properties <- match.arg(properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") type <- match.arg(type, several.ok = TRUE) type <- paste(type, collapse = "|") #Construct URL url <- url_gen(language, project, domain) query_param <- list( action = "query", list = "categorymembers", cmtitle = categories, cmprop = properties, cmtype = type, cmlimit = limit ) #Query and return content <- query(url, "catpages", clean_response, query_param = query_param, ...) return(content) }WikipediR/R/metadata.R0000644000176200001440000002117714604033721014255 0ustar liggesusers#'@title Retrieve a page's backlinks #' #'@description #'page_backlinks, when provided with a page title, retrieves backlinks to that #'page. Output can be filtered to specific namespaces. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param page the title of the page you want the backlinks of. #' #'@param limit the number of backlinks to return. Set to 50 (the maximum) by default. #' #'@param direction the direction to order the backlinks in, by linking page ID: "ascending" #'or "descending". Set to "ascending" by default. #' #'@param namespaces The namespaces to filter to. By default, backlinks from any namespace #'are retrieved: alternately, a numeric vector of accepted namespaces (which are described #'\href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be #'provided, and only backlinks from pages within those namespaces will be returned. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@section Warnings: as with \code{\link{pages_in_category}}, if the page #'you are linking to does not exist, an empty list will be returned, without #'any indication of an error. #' #'@examples #'\dontrun{ #'#Backlink #'all_bls <- page_backlinks("en","wikipedia", page = "Aaron Halfaker") #' #'#Namespace-specific backlinks #'mainspace_bls <- page_backlinks("en","wikipedia", page = "Aaron Halfaker", namespaces = 0) #'} #'@export page_backlinks <- function(language = NULL, project = NULL, domain = NULL, page, limit = 50, direction = "ascending", namespaces = NULL, clean_response = FALSE, ...){ url <- url_gen(language, project, domain) query_param <- list( action = "query", list = "backlinks", bltitle = page, bldir = direction, bllimit = limit ) if(!is.null(namespaces)){ query_param$blnamespace <- paste(namespaces, collapse = "|") } content <- query(url, "blink", clean_response, query_param = query_param, ...) return(content) } #'@title Retrieve a page's links #' #'@description #'page_links, when provided with a page title, retrieves internal wikilinks from the #'current revision of that page. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param page the title of the page you want the links of. #' #'@param limit the number of links to retrieve. 50 by default; a maximum of 500 is set server-side. #' #'@param direction the direction to order the links in, by destination page ID: "ascending" #'or "descending". Set to "ascending" by default. #' #'@param namespaces The namespaces to filter to. By default, links to any namespace #'are retrieved: alternately, a numeric vector of accepted namespaces (which are described #'\href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be #'provided, and only backlinks from pages within those namespaces will be returned. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@examples #'\dontrun{ #'#Links #'links <- page_links("en","wikipedia", page = "Aaron Halfaker") #' #'#Namespace-specific links #'mainspace_links <- page_links("en","wikipedia", page = "Aaron Halfaker", namespaces = 0) #'} #'@export page_links <- function(language = NULL, project = NULL, domain = NULL, page, limit = 50, direction = "ascending", namespaces = NULL, clean_response = FALSE, ...){ url <- url_gen(language, project, domain) query_param <- list( action = "query", prop = "links", titles = page, pldir = direction, pllimit = limit ) if(!is.null(namespaces)){ query_param$plnamespace <- paste(namespaces, collapse = "|") } content <- query(url, "plink", clean_response, query_param = query_param, ...) return(content) } #'@title Retrieve a page's links #' #'@description #'page_external_links, when provided with a page title, retrieves external wikilinks from the #'current revision of that page. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param page the title of the page you want the links of. #' #'@param protocol limit links to those with certain link protocols. Options are listed #'in Special:ApiSandbox's #'\href{https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extlinks}{elprotocol field}. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@examples #'\dontrun{ #'#Links #'external_links <- page_external_links("en","wikipedia", page = "Aaron Halfaker") #' #'#Protocol-specific links #'external_http_links <- page_external_links("en","wikipedia", #' page = "Aaron Halfaker", protocol = "http") #'} #'@export page_external_links <- function(language = NULL, project = NULL, domain = NULL, page, protocol = NULL, clean_response = FALSE, ...){ url <- url_gen(language, project, domain) query_param <- list( action = "query", prop = "extlinks", titles = page ) if(!is.null(protocol)){ query_param$elprotocol <- protocol } content <- query(url, "elink", clean_response, query_param = query_param, ...) return(content) } #'@title Retrieve information about a particular page #' #'@description #'page_info, when provided with a page title, retrieves metadata about that page. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param page the title of the page you want the metadata of. #' #'@param properties the properties you'd like to retrieve. Some properties (the pageID, namespace, #'title, language, length and most recent revision ID, for example) are retrieved by default, #'whatever is passed to \code{properties}: properties that can be explicitly retrieved include #'the page's protection level ("protection"), the ID of the associated talk page, if applicable #'("talkid"), the full, canonical URL ("url"), and the displayed page title ("displaytitle"). #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@examples #'\dontrun{ #'#Metadata #'page_metadata <- page_info("en","wikipedia", page = "Aaron Halfaker") #'} #'@export page_info <- function(language = NULL, project = NULL, domain = NULL, page, properties = c("protection","talkid","url", "displaytitle"), clean_response = FALSE, ...){ properties <- match.arg(arg = properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") url <- url_gen(language, project, domain) query_param <- list( action = "query", prop = "info", inprop = properties, titles = page ) content <- query(url, "pageinfo", clean_response, query_param = query_param, ...) return(content) }WikipediR/R/parse.R0000644000176200001440000000542614601343044013605 0ustar liggesusers#' Response parser #' #' Should not be externally used #' #' @title parse_response: Parse WikipediR responses internally #' @param x result from a WikipediR query #' @export parse_response <- function(x){ UseMethod("parse_response", x) } #' @export parse_response.rchanges <- function(x){ x <- x$query$recentchanges return(x) } #' @export parse_response.rcontent <- function(x){ x <- x$query$pages names(x) <- NULL return(x) } #' @export parse_response.pcontent <- function(x){ x <- x$parse return(x) } #' @export parse_response.rdiff <- function(x){ x <- x$query$pages names(x) <- NULL return(x) } #' @export parse_response.uinfo <- function(x){ x <- x$query$users return(x) } #' @export parse_response.ucontribs <- function(x){ x <- x$query$usercontribs results <- unlist(x) results <- data.frame(matrix(results, nrow = length(x), byrow = TRUE), stringsAsFactors = FALSE) names(results) <- names(x[[1]]) return(results) } #' @export parse_response.catpages <- function(x){ x <- x$query$categorymembers results <- unlist(x) results <- data.frame(matrix(results, nrow = length(x), byrow = TRUE), stringsAsFactors = F) names(results) <- names(x[[1]]) return(results) } #' @export parse_response.pagecats <- function(x){ x <- x$query$pages names(x) <- NULL results <- lapply(x,function(x){ cats <- unlist(x$categories) cats <- data.frame(matrix(cats, nrow = length(x$categories), byrow = TRUE), stringsAsFactors = FALSE) names(cats) <- names(x$categories[[1]]) x$categories <- cats return(x) }) return(results) } #' @export parse_response.blink <- function(x){ x <- x$query$backlinks results <- lapply(x,unlist) return(results) } #' @export parse_response.plink <- function(x){ x <- x$query$pages names(x) <- NULL results <- lapply(x, function(x){ x$links <- lapply(x$links,unlist) return(x) }) return(results) } #' @export parse_response.elink <- function(x){ x <- x$query$pages names(x) <- NULL results <- lapply(x, function(x){ x$extlinks <- unlist(x$extlinks) names(x$extlinks) <- NULL return(x) }) return(results) } #' @export parse_response.pageinfo <- function(x){ x <- x$query$pages names(x) <- NULL results <- lapply(x, function(x){ x$restrictiontypes <- unlist(x$restrictiontypes) return(x) }) return(results) } #' @export parse_response.prelogintoken <- function(x){ x <- x$login$token return(x) } #' @export parse_response.login <- function(x){ x <- x$clientlogin$status == "PASS" return(x) } #' @export parse_response.actiontoken <- function(x){ x <- x$query$tokens$csrftoken return(x) } #' @export parse_response.createpage <- function(x){ x <- x$edit$result == "Success" return(x) }WikipediR/R/login.R0000644000176200001440000000405314601320617013577 0ustar liggesusers#' request token to start client login #' #' helper function to request a user login token #' #' @param url a URL body #' @param user a username of a registered user #' #' @return a token string #' #' @importFrom magrittr "%>%" #' get_prelogin_token <- function(url, user) { # get login token response <- httr::modify_url( url, query = list( action = "login", lgname = user, format = "json" ) ) %>% httr::POST( ) %>% httr::stop_for_status( ) # parse the response, check for API errors parsed_response <- response_parse( response = response, out_class = "prelogintoken" ) if(!is.null(parsed_response$error)){ stop( "The API returned an error: ", parsed_response$error$code, " - ", parsed_response$error$info ) } parse_response(parsed_response) %>% return() } #' wikimedia api user login #' #' Login to a wikimedia instance to trigger api requests #' as a registered user. This function only allows the #' very basic login with username and password. Wikimedia #' setups that require more sophisticated login methods #' are not supported. #' #' @param url a URL body #' @param user a username of a registered user #' @param pw the password of said user #' #' @return TRUE #' #' @export login <- function(url, user, pw){ # get prelogin token token <- get_prelogin_token(url, user) # login response <- httr::modify_url( url, query = list( action = "clientlogin", username = user, rememberMe = 1, loginreturnurl = url, format = "json" ) ) %>% httr::POST( body = list( logintoken = token, password = pw ) ) %>% httr::stop_for_status( ) # parse the response, check for API errors parsed_response <- response_parse( response = response, out_class = "login" ) if(!is.null(parsed_response$error)){ stop( "The API returned an error: ", parsed_response$error$code, " - ", parsed_response$error$info ) } parse_response(parsed_response) %>% return() } WikipediR/R/recent_changes.R0000644000176200001440000000747414601320617015451 0ustar liggesusers#'@title Retrieves entries from the RecentChanges feed #' #'@description #'wiki_recentchanges retrieves a stream of entries from Special:RecentChanges, with a variety of #'associated metadata and filtering (of both entries *and* that metadata. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param properties Properties you're trying to retrieve about each entry, Options include #'"user" (the username of the person responsible for that entry), "userid" (the userID of said #'person), "comment" (the edit summary associated with the entry), "parsedcomment" (the same, #'but parsed, generating HTML from any wikitext in that comment), "flags" (whether the revision #'was 'minor' or not), "timestamp", "title" (the name of the page the entry affected), "ids" #'(the page id, along with the old and new revision IDs when applicable) "sizes" (the size, #'in uncompressed bytes, of the entry, and, in the case of revisions, the size of the edit #'it displaced), "tags" (any tags associated with the revision) and "loginfo" (applicable only #'to log entries, and consisting of log ID numbers, log types and actions, and so on) and "sha1" #'(the SHA-1 hash of the revision text). #' #'@param type The type of entry you want to retrieve; can be any permutation of "edit" (edits to existing pages), #'"external" (external actions that impact on the project - primarily wikidata changes), #'"new" (the creation of new pages) and "log" (log entries). By default, all of these entry types #'are included. #' #'@param tag Only return items with particular "tags", such as "mobile edit". NULL by #'default. #' #'@param dir Should it go from newest to oldest ("newer"), or oldest to newest ("older")? #'By default, set to "newer". #' #'@param limit The number of entries you'd like to return. By default, set to 50, which is #'also the maximum number per-request for logged-out users. #' #'@param top Should the request only return "top" entries - in other words, the most recent #'entry on a page? Set to FALSE by default. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@export recent_changes <- function(language = NULL, project = NULL, domain = NULL, properties = c("user","userid","comment", "parsedcomment","flags","timestamp", "title","ids","sizes","redirect", "loginfo","tags","sha1"), type = c("edit","external","new","log"), tag = NULL, dir = "newer", limit = 50, top = FALSE, clean_response = FALSE, ...) { #Format and standardise, construct URL type <- match.arg(arg = type, several.ok = TRUE) type <- paste(type, collapse = "|") properties <- match.arg(arg = properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") url <- url_gen(language, project, domain) query_param <- list( action = "query", list = "recentchanges", rcdir = dir, rcprop = properties, rctype = type, rclimit= limit ) if(!is.null(tag)){ query_param$rctag <- paste(tag, collapse = "|") } if(top){ query_param$rctoponly <- "" } #Query content <- query(url, "rchanges", clean_response, query_param = query_param, ...) #Return return(content) }WikipediR/R/generic_handlers.R0000644000176200001440000000122714601320617015763 0ustar liggesusers#Handles limitations on the number of unique values you can pass in. #Takes an input set of params and a limit, and warns of the length of one #is greather than the other, limiting the set if so, and either way, #collapsing for incorporation into the API query. handle_limits <- function(parameters, limit){ if(length(parameters) > limit){ warning("This option accepts ", limit, " values; you have provided ", length(parameters), ". Only the first ", limit, " will be returned.", call. = FALSE) parameters <- paste(parameters[1:limit], collapse = "|") } else { parameters <- paste(parameters, collapse = "|") } return(parameters) }WikipediR/R/content.R0000644000176200001440000003162614604033647014156 0ustar liggesusers#Checks for invalid revision IDs, warns if they're found. invalid_revs <- function(parsed_response){ if(!is.null(parsed_response$query$badrevids)){ warning("This request contained ",length(parsed_response$query$badrevids)," invalid revisionID(s)", call. = FALSE) } return(invisible()) } #'@title Retrieve the page content of a random MediaWiki page #' #'@description #'wiki_page retrieves the DOM of a particular MediaWiki page, #'as a HTML blob inside a JSON object. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param namespaces The namespaces to consider pages from. By default, pages from any namespace are #'considered; alternately, a numeric vector of accepted namespaces (which are described #'\href{https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces}{here}) can be #'provided, and only pages within those namespaces will be considered. #' #'@param as_wikitext whether to retrieve the wikimarkup (TRUE) or the HTML (FALSE). #'Set to FALSE by default. #' #'@param limit the number of pages to return. 1 by default. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@seealso \code{\link{page_content}} for retrieving the content of a specific page, #'\code{\link{revision_diff}} for retrieving 'diffs' between revisions, #'\code{\link{revision_content}} for retrieving the text of specified revisions. #' #'@examples #'\dontrun{ #'#A page from Wikipedia #'wp_content <- random_page("en","wikipedia") #' #'#A page from the mainspace on Wikipedia #'wp_article_content <- random_page("en","wikipedia", namespaces = 0) #'} #'@export random_page <- function(language = NULL, project = NULL, domain = NULL, namespaces = NULL, as_wikitext = FALSE, limit = 1, clean_response = FALSE, ...){ url <- url_gen(language, project, domain) query_param <- list( action = "query", list = "random", rnlimit = limit ) if(!is.null(namespaces)){ query_param$rnnamespace <- paste(namespaces, collapse = "|") } pages <- query(url, NULL, FALSE, query_param = query_param)$query$random return(lapply(pages, function(page, language, project, domain, page_name, as_wikitext, clean_response, ...){ content <- page_content(language = language, project = project, domain = domain, page_name = page$title, as_wikitext = as_wikitext, clean_response = clean_response, ...) content$parse$text <- content$parse$text[[1]] return(content$parse) }, language = language, project = project, domain = domain, as_wikitext = as_wikitext, clean_response = clean_response, ...)) } #'@title Retrieves MediaWiki page content #' #'@description #'wiki_page retrieves the DOM of a particular MediaWiki page, #'as a HTML blob inside a JSON object. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param page_name The title of the page you want to retrieve #' #'@param page_id the pageID of the page you want to retrieve. Set to NULL by default, #'and an alternative to page_name; if both are provided, page_id will be used. #' #'@param as_wikitext whether to retrieve the wikimarkup (TRUE) or the HTML (FALSE). #'Set to FALSE by default. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@seealso \code{\link{revision_diff}} for retrieving 'diffs' between revisions, #'\code{\link{revision_content}} for retrieving the text of specified revisions. #' #'@examples #'\dontrun{ #'#Content from a Wikimedia project #'wp_content <- page_content("en","wikipedia", page_name = "Aaron Halfaker") #' #'#Content by ID #'wp_content <- page_content("en", "wikipedia", page_id = 12) #' #'#Content from a non-Wikimedia project #'rw_content <- page_content(domain = "rationalwiki.org", page_name = "New Age") #'} #'@export page_content <- function(language = NULL, project = NULL, domain = NULL, page_name, page_id = NULL, as_wikitext = FALSE, clean_response = FALSE, ...){ #Format and construct URL. if(as_wikitext){ properties <- "wikitext|revid" } else { properties <- "text|revid" } properties <- paste(properties, collapse = "|") url <- url_gen(language, project, domain) query_param <- list( action = "parse", prop = properties ) if(!is.null(page_id)){ query_param$pageid <- handle_limits(page_id, 1) } else { query_param$page <- handle_limits(page_name, 1) } #Run content <- query(url, "pcontent", clean_response, query_param = query_param, ...) #Return return(content) } #'@title Retrieves MediaWiki revisions #' #'@description #'Retrieves the content of a provided list of revisions from whichever MediaWiki instance you're #'querying. Returns as wikimarkup. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param revisions The revision IDs of each desired revision. #' #'@param properties Properties you're trying to retrieve about that revision, should you want to; #'options include "ids" (the revision ID of the revision...which is pointless), #'"flags" (whether the revision was 'minor' or not), "timestamp" (the timestamp of the revision), #'"user" (the username of the person who made that revision), "userid" #'(the userID of the person who made the revision), #'"size" (the size, in uncompressed bytes, of the revision), "sha1" (the SHA-1 hash of #'the revision text), "contentmodel" (the content model of the page, usually "wikitext"), #'"comment" (the revision summary associated with the revision), "parsedcomment" (the same, #'but parsed, generating HTML from any wikitext in that comment), "tags" (any tags associated #'with the revision) and "flagged" (the revision's status under Flagged Revisions). #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #' #'@param ... further arguments to pass to httr's GET. #' #'@seealso #'\code{\link{revision_diff}} for diffs between revisions, #'and \code{\link{page_content}} for the content a specific page currently has. #' #'@examples #'\dontrun{ #'#Revision content from a Wikimedia project #'wp_content <- revision_content("en","wikipedia", revisions = 552373187) #' #'#Revision content from a non-Wikimedia project #'rw_content <- revision_content(domain = "rationalwiki.org", revisions = 88616) #'} #'@export revision_content <- function(language = NULL, project = NULL, domain = NULL, revisions, properties = c("content","ids","flags","timestamp", "user","userid","size", "sha1","contentmodel","comment", "parsedcomment","tags"), clean_response = FALSE, ...){ #Format, construct URL. properties <- match.arg(arg = properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") revisions <- handle_limits(revisions, 50) url <- url_gen(language, project, domain) query_param <- list( rvcontentformat = "text/x-wiki", action = "query", prop = "revisions", rvprop = properties, revids = revisions ) #Run content <- query(url, "rcontent", clean_response, query_param = query_param, ...) #Check for invalid RevIDs invalid_revs(content) #Return return(content) } #'@title Generates a "diff" between a pair of revisions #' #'@description #'revision_diff generates a diff between two revisions in a MediaWiki page. #'This is provided as an XML-parsable blob inside the returned JSON object. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param revisions The revision IDs of each "start" revision. #' #'@param properties Properties you're trying to retrieve about that revision, should you want to; #'options include "ids" (the revision ID of the revision...which is pointless), #'"flags" (whether the revision was 'minor' or not), "timestamp","user" (the username of the person #'who made that revision), "userid" (the userID of the person who made the revision), #'"size" (the size, in uncompressed bytes, of the revision), "sha1" (the SHA-1 hash of #'the revision text), "contentmodel" (the content model of the page, usually "wikitext"), #'"comment" (the revision summary associated with the revision), "parsedcomment" (the same, #'but parsed, generating HTML from any wikitext in that comment), "tags" (any tags associated #'with the revision) and "flagged" (the revision's status under Flagged Revisions). #' #'@param direction The direction you want the diff to go in from the revisionID you have provided. #'Options are "prev" (compare to the previous revision on that page), "next" (compare to the next #'revision on that page) and "cur" (compare to the current, extant version of the page). #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #' #'@param ... further arguments to pass to httr's GET. #' #'@section Warnings: #' #'MediaWiki's API is deliberately designed to restrict users' ability to make computing-intense requests #'- such as diff computation. As a result, the API only allows requests for one uncached diff in #'each request. If you ask for multiple diffs, some uncached and some cached, you will be provided #' with the cached diffs, one of the uncached diffs, and a warning. #' #'If you're going to be asking for a lot of diffs, some of which may not be cached, it may be more #'sensible to retrieve the revisions themselves using \code{\link{revision_content}} and compute the #'diffs yourself. #' #'@seealso \code{\link{page_content}} for retrieving the current content of a specific page, and #'\code{\link{revision_content}} for retrieving the text of specific revisions. #' #'@examples #'\dontrun{ #'#Wikimedia diff #'wp_diff <- revision_diff("en","wikipedia", revisions = 552373187, direction = "next") #' #'#Non-Wikimedia diff #'rw_diff <- revision_diff(domain = "rationalwiki.org", revisions = 88616, direction = "next") #'} #'@export revision_diff <- function(language = NULL, project = NULL, domain = NULL, revisions, properties = c("ids","flags","timestamp","user","userid","size", "sha1","contentmodel","comment","parsedcomment", "tags","flagged"), direction, clean_response = FALSE, ...){ #Check and construct URL direction <- match.arg(direction, c("prev","next","cur")) properties <- match.arg(properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") revisions <- handle_limits(revisions, 50) url <- url_gen(language, project, domain) query_param <- list( action = "query", prop = "revisions", rvprop = properties, rvdiffto = direction, rvcontentformat = "text/css", revids = revisions ) #Retrieve the content, check for invalid RevIDs and uncached diffs, #return. content <- query(url, "rdiff", clean_response, query_param = query_param, ...) invalid_revs(content) if(sum(grepl(x = names(unlist(content)), pattern = "diff.notcached"))){ warning("This request contained uncached diffs; these will not be returned", call. = FALSE) } return(content) }WikipediR/R/create_content.R0000644000176200001440000000602214601320617015462 0ustar liggesusers#' request token for api action as signed in user #' #' helper function to request a user action token #' #' @param url a URL body #' #' @return a token string #' #' @importFrom magrittr "%>%" #' get_action_token <- function(url) { # get login token response <- httr::modify_url( url, query = list( action = "query", meta = "tokens", format = "json" ) ) %>% httr::GET( ) %>% httr::stop_for_status( ) # parse the response, check for API errors parsed_response <- response_parse( response = response, out_class = "actiontoken" ) if(!is.null(parsed_response$error)){ stop( "The API returned an error: ", parsed_response$error$code, " - ", parsed_response$error$info ) } parse_response(parsed_response) %>% return() } #' wikimedia api page creation #' #' Create pages or category-pages on a wikimedia instance. #' #' @param url a URL body #' @param p_title vector with page title strings of new pages #' @param p_text vector with page content strings of new pages #' @param category switch to decide, if the pages should be #' created as category-pages #' #' @return TRUE #' #' @export create_pages <- function(url, p_title, p_text, category = FALSE){ if(length(p_title) != length(p_text)) { stop( "The length of the vectors p_title and p_text is not equal." ) } # get action token token <- NA token <- get_action_token(url) if(token %>% is.na){ stop("Problems with the action token request.") } # create page(s) (with progress bar) pb <- utils::txtProgressBar( min = 0, max = length(p_title), style = 3 ) res <- c() for (i in 1:length(p_title)) { res[i] <- create_page( url, p_title[i], p_text[i], category, token ) utils::setTxtProgressBar(pb, i) } close(pb) all(res) %>% return() } #' wikimedia api page creation (single pages) #' #' helper function to do the actual api requests for page #' and category-page creation #' #' @param url a URL body #' @param p_title page title string of new page #' @param p_text page content string of new page #' @param category switch to decide, if the page should be #' created as category-page #' @param token action token to perform the request #' #' @return TRUE #' create_page <- function(url, p_title, p_text, category, token){ # create page response <- httr::modify_url( url, query = list( action = "edit", title = ifelse( category, paste0("Category:", p_title), p_title ), text = p_text, format = "json" ) ) %>% httr::POST( body = list(token = token) ) %>% httr::stop_for_status( ) # parse the response, check for API errors parsed_response <- response_parse( response = response, out_class = "createpage" ) if(!is.null(parsed_response$error)){ stop( "The API returned an error: ", parsed_response$error$code, " - ", parsed_response$error$info ) } parse_response(parsed_response) %>% return() } WikipediR/R/query.R0000644000176200001440000000471114601320617013635 0ustar liggesusers#General functions and error handlers for #generic queries and query construction. #'@title base query function #'@description not designed to be used by anyone except #'a third-party reuser package, such as WikidataR #'@param url a URL body #'@param out_class the class to set on the output object; used within #'WikidataR to indicate what response-cleaning method should be applied. #' #'@param clean_response whether to clean the response, using the method assigned #'by out_class, or not. #' #'@param query_param query parameters #' #'@param ... further arguments to httr's GET. #'@export query <- function(url, out_class, clean_response = FALSE, query_param = list(), ...){ # Common query parameters if(is.null(query_param$format)) { query_param$format <- "json" } args <- list(...) if(length(args) > 0 && "config" %in% class(args[[1]]) && "useragent" %in% names(args[[1]])){ response <- httr::GET(url, query = query_param, ...) } else { response <- httr::GET(url, query = query_param, httr::user_agent("WikipediR - https://github.com/Ironholds/WikipediR"), ...) } #Check the validity of the response httr::stop_for_status(response) #Parse the response, check for API errors, return parsed_response <- response_parse(response = response, out_class = out_class) if(!is.null(parsed_response$error)){ stop("The API returned an error: ", parsed_response$error$code, " - ", parsed_response$error$info) } if(clean_response){ parsed_response <- parse_response(parsed_response) } return(parsed_response) } #Constructor for the URL url_gen <- function(language, project, domain = NULL, ...){ if(is.null(domain)){ #Commons and Wikispecies have different URL formats, so those have to be handled in a hinky way. if(project %in% c("commons","species")){ url <- sprintf("https://%s.wikimedia.org/w/api.php", project) } else { url <- sprintf("https://%s.%s.org/w/api.php", language, project) } } else { url <- sprintf("http://%s/w/api.php", domain) } #Return return(url) } #'@importFrom httr content #'@importFrom jsonlite fromJSON response_parse <- function(response, out_class){ #Convert it into a character vector response_text <- httr::content(x = response, as = "text") #From there, turn it into an R object from JSON parsed_text <- jsonlite::fromJSON(txt = response_text, simplifyVector = FALSE) class(parsed_text) <- out_class #Return return(parsed_text) } WikipediR/R/WikipediR.R0000644000176200001440000000110614601321137014350 0ustar liggesusers#' @title A client library for MediaWiki's API #' @name WikipediR #' @description This package provides functions for accessing the MediaWiki API, either for #' Wikimedia projects or any other MediaWiki instance. For more information, see the #' \href{https://CRAN.R-project.org/package=WikipediR/vignettes/WikipediR.html}{vignette}. #' #' @seealso The \href{https://CRAN.R-project.org/package=WikipediR/vignettes/WikipediR.html}{package vignette}. #' @importFrom httr GET user_agent stop_for_status #' @importFrom utils URLencode #' @aliases WikipediR WikipediR-package "_PACKAGE"WikipediR/R/user_info.R0000644000176200001440000002102014604033757014462 0ustar liggesusers#Missing user handler for user_information/user_contributions missing_users <- function(parsed_response){ #Check for missing values names_to_check <- names(unlist(parsed_response)) missing_names <- sum(grepl(x = names_to_check, pattern = "users\\.missing")) if(missing_names){ warning(missing_names," of the provided usernames did not exist", call. = FALSE) } return(invisible()) } #'@title Retrieve user contributions #' #'@description Retrieves metadata associated with the most recent contributions by a #'specified user. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param username The username of the user whose contributions you want to retrieve. #'Due to limitations at the API end, you can only retrieve edits for one user at a time. #' #'@param properties The metadata you want associated with each edit. Potential metadata includes "ids" #'(the revision ID of the revision, which can be passed into \code{\link{revision_content}}), #'"title" (the name of the page that was edited), "timestamp", "comment" (the edit summary associated #'with the revision), "parsedcomment" (the same, but parsed, generating HTML from any wikitext #'in that comment), "size" (the size, in uncompressed bytes, of the edit), "sizediff" (the size #'delta between this edit, and the last edit to the page), "flags" (whether the revision was #''minor' or not), and "tags" (any tags associated with the revision). #' #'@param mainspace A boolean flag; FALSE retrieves all of the most recent contributions, while #'TRUE limits the retrieved contributions to those in the 'mainspace' - in other words, edits to #'actual articles. Set to FALSE by default #' #'@param limit The number of edits to be retrieved. 50 is the maximum for logged-out API users, #'and putting in more than 50 will generate a warning. #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@seealso \code{\link{user_information}} for information about a specific user (or group of users), #'and \code{recent_changes} for non-user-specific recent actions. #' #'@examples #'\dontrun{ #'#Retrieve the timestamps of a user's recent contributions to the English-language Wikipedia #'contribs <- user_contributions("en", "wikipedia", username = "Ironholds", #' properties = "timestamp") #' #'#Retrieve the timestamps of a user's recent contributions to a non-Wikimedia wiki. #'rw_contribs <- user_contributions(domain = "rationalwiki.org", username = "David Gerard", #' properties = "ids", limit = 1) #'} #'@export user_contributions <- function(language = NULL, project = NULL, domain = NULL, username, properties = c("ids", "title", "timestamp", "comment", "parsedcomment", "size", "sizediff", "flags", "tags"), mainspace = FALSE, limit = 50, clean_response = FALSE, ...){ #Perform checks, construct URL properties <- match.arg(properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") username <- handle_limits(username, 1) url <- url_gen(language, project, domain) query_param <- list( action = "query", list = "usercontribs", uclimit = limit, ucuser = username, ucprop = properties ) #If only article contributions are desired, note that. if(mainspace){ query_param$ucnamespace <- 0 } #Get, check and return contribs_content <- query(url, "ucontribs", clean_response, query_param = query_param, ...) missing_users(contribs_content) return(contribs_content) } #'@title Retrieve user information #' #'@description #'Retrieves information about a user, or set of users, from the MediaWiki API, #'including registration date, gender and editcount. #' #'@param language The language code of the project you wish to query, #'if appropriate. #' #'@param project The project you wish to query ("wikiquote"), if appropriate. #'Should be provided in conjunction with \code{language}. #' #'@param domain as an alternative to a \code{language} and \code{project} combination, #'you can also provide a domain ("rationalwiki.org") to the URL constructor, allowing #'for the querying of non-Wikimedia MediaWiki instances. #' #'@param user_names The username(s) of the users you want information on - this should be provided #'as a vector. There is a hard limit of 50 distinct users per query, set by MediaWiki's API; #'in the event that you go over this, a warning will be issued and the query will only be #'performed for the first 50 names in the vector. #' #'@param properties The user properties you're interested in. Applicable properties are #'"blockinfo" (details about the user's block, if they are currently blocked), "groups" #'(the user groups the user is a member of), "implicitgroups" (groups they are a member of #'through inheritance, as a result of membership in other groups), "rights" (what permissions #'their group membership grants them), "editcount" (how many non-deleted edits they have), #'"registration" (the date when they registered), "emailable" (whether they are contactable #'through Special:EmailUser) and "gender" (their provided gender). #' #'@param clean_response whether to do some basic sanitising of the resulting data structure. #'Set to FALSE by default. #' #'@param ... further arguments to pass to httr's GET. #' #'@section Warnings: #'There are a few caveats with the data provided by \code{user_information}, mostly stemming from #'historical inconsistencies and peculiarities in MediaWiki. #' #'\code{groups} and \code{implicitgroups} gives you the user's permissions and group membership #'on the project you are querying, not their membership on all projects - while you can find out #'if "Ironholds" is not a sysop on, say, enwiki, that doesn't mean they aren't a sysop elsewhere #'- there is no universal, API-accessible user groups listing. #' #'As an extension of the lack of centrality in Wikimedia's infrastructure, \code{registration} #'tells you the date their account was created on the wiki you are querying. If they initially #'registered on that wiki, this is accurate - if they registered on a different wiki, #'this instead reflects the date and time that they first visited the wiki you're querying #'while logged-in. For users registered before 2006, when registration logging was introduced, #'the \code{registration} value represents not when they first registered, but when their first #'edit was, since that was used as an estimator for existing accounts when the field was first #'populated. #' #'@seealso \code{\link{user_contributions}} for retrieving recent contributions made by #'a particular user. #' #'@examples #'\dontrun{ #'#Retrieving information from a Wikimedia project #'user_info <- user_information("en", "wikipedia", user_names = "David Gerard", #' properties = "registration") #' #'#Non-Wikimedia projects #'user_info <- user_information(domain = "rationalwiki.org", user_names = "David Gerard", #' properties = "registration") #'} #'@export user_information <- function(language = NULL, project = NULL, domain = NULL, user_names, properties = c("blockinfo","groups","implicitgroups", "rights","editcount","registration", "emailable","gender"), clean_response = FALSE, ...){ #Check, construct URL properties <- match.arg(properties, several.ok = TRUE) properties <- paste(properties, collapse = "|") user_names <- handle_limits(user_names, 50) url <- url_gen(language, project, domain) query_param = list( action = "query", list = "users", usprop = properties, ususers = user_names ) #Retrieve the content, check it, return. user_content <- query(url, "uinfo", clean_response, query_param = query_param, ...) missing_users(user_content) return(user_content) }WikipediR/MD50000644000176200001440000000465514604037454012472 0ustar liggesusers8a1187bc162bec655cd593c2e9780e60 *DESCRIPTION 1d9678dbfe1732b5d2c521e07b2ceef0 *LICENSE 1aa74073b35fca87f224628c2c05b2df *NAMESPACE 4f85f4bc8d56477833d6e5518a0b93a5 *NEWS 7abe9dbaf67fb6edb3d06f7e431457d2 *R/WikipediR.R 5067ad5283bcd433dba907eea0796562 *R/categories.R 73503b4191ac966ca97186c30c9e4306 *R/content.R 4bcf6a5b0429c26143f57a5754cc181a *R/create_content.R 3310e6008ce3c185829aa4fe5087baf2 *R/generic_handlers.R 83bb45ed7646a35c5a067aa48b86af4b *R/login.R 77d1566ddebc8dd53c5d4d5ffe44be99 *R/metadata.R b28c261d06ea9722155ba633658d76c7 *R/parse.R 7d6b7516a10fa66ee830d20dcde26594 *R/query.R 9e782c267ce639c3c333b567b93bf8d8 *R/recent_changes.R cbf979fa8a25eabe82bdcec7584afebc *R/user_info.R 4ff28c992f97f12f9d2d1a584a1f7996 *README.md 8427e4d2e5c7326c57c871aff38d68cc *build/vignette.rds e5214179299ab80dd7b16e98346cd8f1 *inst/doc/WikipediR.Rmd 8bd185cd6e77ed314d9c518ca3418dcc *inst/doc/WikipediR.html 61ea5b5e2197b3e95cda09ba8026bc8b *man/WikipediR.Rd bc835670547b03a522697e7bbc37bc91 *man/categories_in_page.Rd 87376fcad8f502c1e5096509c7dce3ea *man/create_page.Rd a6047d320bc1e72988140343ec07c28f *man/create_pages.Rd 835829e6cae2db0512fb8642c605e5e1 *man/get_action_token.Rd 55b22500bc5ad9b135e3d9d0049efa15 *man/get_prelogin_token.Rd 43294b43d3f0beb196af50d47cd0b5d1 *man/login.Rd b5f06d794c1980deb7a2d13d52435a91 *man/page_backlinks.Rd 08e4b35bfefb7e8f7d76be7006ac876e *man/page_content.Rd baf00e4335d0d7bc0d02305af0c19e35 *man/page_external_links.Rd 646476cdcfb2cb2fff2f7f4a60c6c82c *man/page_info.Rd 7d79d48d4f581db9d41c98eb1bbf5657 *man/page_links.Rd eafb1a42cdec850391c0530a1c3daf00 *man/pages_in_category.Rd f488e4051f977b098ea337ff8479542b *man/parse_response.Rd a81ee6a673391569ebfcf7869f7a59fa *man/query.Rd 2f9ff7f1dc98a584748f2744a3d56190 *man/random_page.Rd c2c0e8414545359e45ef101cd8531991 *man/recent_changes.Rd 36b024ad0f73ebbe9d7195b37d2e59d6 *man/revision_content.Rd ca9c157a8432d70d4043d983cdf6a54d *man/revision_diff.Rd e3457b8cba3674b546fa670db8d11073 *man/user_contributions.Rd 9a4f5ff01273099b0a5691383d4c19c7 *man/user_information.Rd 0258bf928f7abd1db4cf997cbff76363 *tests/testthat.R 700d7e68d80d76edb1f345f679aef317 *tests/testthat/test_category_retrieval.R e5113c7e7fe442a49f79492861102792 *tests/testthat/test_content_retrieval.R 82c679290bcde1b558517bf945cb41ae *tests/testthat/test_metadata_retrieval.R fe113c80acc16e643f44d946c03927c1 *tests/testthat/test_recent_changes.R e5214179299ab80dd7b16e98346cd8f1 *vignettes/WikipediR.Rmd WikipediR/inst/0000755000176200001440000000000014604034112013111 5ustar liggesusersWikipediR/inst/doc/0000755000176200001440000000000014604034112013656 5ustar liggesusersWikipediR/inst/doc/WikipediR.html0000644000176200001440000001035414604034112016436 0ustar liggesusers Retrieving content

#WikipediR: A MediaWiki API client library Many websites run on versions of MediaWiki, most prominently Wikipedia and its sister sites. WikipediR is an API client library that allows you to conveniently make requests for content and associated metadata against MediaWiki instances.

Retrieving content

“content” can mean a lot of different things - but mostly, we mean the text of an article, either its current version or any previous versions. Current versions can be retrieved using page_content, which provides both HTML and wikitext as possible output formats. Older, individual revisions can be retrieved with revision_content. These functions also return a range of possible metadata about the revisions or articles in question.

Diffs between revisions can be generated using revision_diff, while individual ''elements'' of a page's content - particularly links - can be extracted using page_links, page_backlinks, and page_external_links. And if the interest is in changes to content, rather than content itself, recent_changes can be used to grab a slice of a project's Special:RecentChanges feed.

Retrieving metadata

Page-related information can be accessed using page_info, while categories that a page possesses can be retrieved with categories_in_page - the inverse of this operation (what pages are in a particular category?) uses pages_in_category.

User-related info can be accessed with user_information, while user_contributions allows access to recent contributions by a particular user: this can be conveniently linked up with revision_content, mentioned above, to retrieve the content of the last N edits by a particular editor, or metadata about those edits.

WikipediR/inst/doc/WikipediR.Rmd0000644000176200001440000000363214601320617016222 0ustar liggesusers #WikipediR: A MediaWiki API client library Many websites run on versions of MediaWiki, most prominently Wikipedia and its sister sites. WikipediR is an API client library that allows you to conveniently make requests for content and associated metadata against MediaWiki instances. ## Retrieving content "content" can mean a lot of different things - but mostly, we mean the text of an article, either its current version or any previous versions. Current versions can be retrieved using page\_content, which provides both HTML and wikitext as possible output formats. Older, individual revisions can be retrieved with revision\_content. These functions also return a range of possible metadata about the revisions or articles in question. Diffs between revisions can be generated using revision\_diff, while individual ''elements'' of a page's content - particularly links - can be extracted using page\_links, page\_backlinks, and page\_external\_links. And if the interest is in changes to content, rather than content itself, recent\_changes can be used to grab a slice of a project's Special:RecentChanges feed. ## Retrieving metadata Page-related information can be accessed using page\_info, while categories that a page possesses can be retrieved with categories\_in\_page - the inverse of this operation (what pages are in a particular category?) uses pages\_in\_category. User-related info can be accessed with user\_information, while user\_contributions allows access to recent contributions by a particular user: this can be conveniently linked up with revision\_content, mentioned above, to retrieve the content of the last N edits by a particular editor, or metadata about those edits.