Skip to content

Platforms

Supported Platforms

Get a list of supported platforms with Skyscraper --help.

After the initial work from torresflo @ GitHub it is possible to add new platforms by editing the platforms.json file. However, since version 3.9.0 of Skyscraper this file is replaced by peas.json (short for platforms, extensions/formats, aliases and scrapers. The scrapers list has been removed with 3.13.0 as it did not have any use) in the same folder.

Take this example from the peas.json file:

    [...]
    "megadrive": {
        "aliases": [
            "genesis",
            "mega drive",
            "sega mega drive",
            "sega mega drive/genesis"
        ],
        "formats": [
            "*.bin",
            "*.gen",
            "*.md",
            "*.sg",
            "*.smd"
        ]
    },
    [...]
  • <key>: the key (megadrive in the example) reflects the platform/folder name (or more generally the handle), usually provided with -p on the command line.
  • aliases: these aliases can be used to allow additional hits for a search. The use of aliases is handy especially for scraping websites which do not provide a web API. Currently Screenscraper, Mobygames or The Games DB are providing an web API. See also section about platforms_idmap.csv and Updating the files below.
  • formats: set of ROM file extensions which will be included in scraping if a ROM file is not provided explicitly via command line.

Tip

Since Skyscraper 3.13.0 you may also maintain local changes to the peas.json in a separate file called peas_local.json alongside to the peas.json. The format is identical to the peas.json. Precedence: Any platform entry from peas_local.json with the same platform name as in peas.json overwrites the one from peas.json.

Tip

If you need a specific folder name for a platform (on your setup or due to an EmulationStation theme) use a symbolic link (see megadrive (=folder) and genesis (=symlink) for example on RetroPie or plus4 (=folder) and c16 (=symlink)) instead of adding a new platform in the JSON file.

Exact platform mapping

The file platforms_idmap.csv defines exact platform id for the web API of the three before mentioned scraping sites. It is a CSV file which maps the platform handle (e.g. megadrive) to the respective platform id of the scraping sites:

folder,screenscraper_id,mobygames_id,tgdb_id
[...]
megadrive,1,16,36
[...]

You can display the number with their platform name on each of the three scraping sites with the script peas_and_idmap_verify.py. Find the script sibling to the Skyscraper executable. Below is a part of the output:

[...]
    ├── megadrive
    │   ├── ScrS    1: Megadrive
    │   ├── Moby   16: Genesis
    │   └── TGDB   36: Sega Mega Drive
[...]

How to Add Platforms For Scraping

Outline:

  1. Create a file peas_local.json sibling to peas.json. Enter in this file an empty {} JSON object.
  2. Create a new platform block in peas_local.json inside the outer (empty) block created before, or copy an existing block and adapt to your needs. For RetroPie your chosen <platform_name> must match the folder in ~/RetroPie/roms/<platform_name>.
  3. Use <platform_name> also in platforms_idmap.csv (see below for details).
  4. If you use RetroPie do add the platform/system also to your es_systems.cfg as documented here.

There is also a an verbatim example, you may skip the next section initially and can continue with the hands-on example.

Updating peas_local.json and platforms_idmap_local.csv

These two files are ment to be locally edited and extended for additional platforms. Whenever you add a new platform block to the peas_local.json do also lookup the corresponding platform ids and add them to platforms_idmap_local.csv for the scraping sites with an API.

From Skyscraper 3.15.0 onwards creating/editing the peas_local.json and platforms_idmap_local.csv is the preferred way. In any case both files (e.g. peas.json and peas_local.json) will be evalutated but the _local configuration has precedence over the distributed peas.json. The same rule applies to platforms_idmap.csv and platforms_idmap_local.csv.
If you have made local changes before Skyscraper 3.15 to either peas.json or platforms_idmap.csv read the section Transferring Local Platform Changes on how to transfer your changes to the corresponding *_local.* files.

To find the platform ids for Screenscraper, Mobygames and The Games DB, please consult the files screenscraper_platforms.json, mobygames_platforms.json and tgdb_platforms.json which are located sibling to your config.ini of the Skyscraper installation. If you can not identify an id in these files use -1 as value in the CSV. If you add -1 to CSV, the aliases from peas are tried to find a match upon scraping. Edits in screenscraper_platforms.json, mobygames_platforms.json and tgdb_platforms.json are not needed and moreover they will be overwritten with each Skyscraper update, as these files are only a reference for finding the id values for the platforms_idmap.csv.

For those scraping sites without an API or without exact id match do use the platform name which is used on the scraping site and put it into the in the aliases list in the peas.json for the respective platform/system at <platform_name>.

Example

The platforms ScummVM or Steam do not have an exact match on Mobygames, however you may scrape successfully for ScummVM and Steam games if you use 'PC', 'DOS', 'Windows', 'Linux' or similar as "aliases": ... in the "scummvm": ... or "steam": ... section of peas.json. Usually you find the platform information if you lookup the game manually on the scraping website.

Sample Usecase: Adding Platform Satellaview

Let the platform/systemname be satellaview. You may read about this SNES enhancing peripheral here.

Step 1: Add a Section to peas_local.json

Actually, this is a copy of the "snes: ..." block, with slight modifications.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
...
"satellaview": {
    "aliases": [
        "snes",
        "nintendo power",
        "snes - super mario world hacks",
        "sufami turbo",
        "super famicom",
        "super nintendo",
        "super nintendo (snes)",
        "super nintendo entertainment system (snes)",
        "super nintendo msu-1"
    ],
    "formats": [
        "*.bin",
        "*.fig",
        "*.mgd",
        "*.sfc",
        "*.smc",
        "*.swc"
    ]
}
...
  • Line 2 defines the platform name
  • Line 4 adds an alias to SNES as Satellaview is an hardware addon to a SNES

Step 2: Lookup Values for Satellaview and Update platforms_idmap.csv

To fill in the values into the CSV file do consult the screenscraper_platforms.json, mobygames_platforms.json and tgdb_platforms.json files. Try to find the ID for the platform in these files. For satellaview only Screenscraper has an exact match: 107. Use -1 as ID when no exact match is provided for the scraping site. Whenever an -1 is encountered Skyscraper tries the "aliases": from peas.json (and from peas_local.json if present) to find scraping data.

Add this information at the end of the CSV file. Why at the end? On rare occasions the platforms_idmap.csv may be updated on a new release. On RetroPie the installed file is then named platforms_idmap.csv.rp-dist, having changes at the end may be a less cumbersome manual merge with your local platforms_idmap.csv.

Add this information:

satellaview,107,-1,-1

Step 3: Create the System in RetroPie/EmulationStation and Populate the ROM Folder

This part should be added to your ~/.emulationstation/es_systems.cfg. See here for additional information on this.

Warning

Stop EmulationStation before editing the config file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
...
<system>
    <name>satellaview</name>
    <fullname>Satellaview</fullname>
    <path>/home/pi/RetroPie/roms/Nintendo - Satellaview</path>
    <extension>.7z .bin .bs .smc .sfc .fig .swc .mgd .zip .7Z .BIN .BS .SMC .SFC .FIG .SWC .MGD .ZIP</extension>
    <command>/opt/retropie/supplementary/runcommand/runcommand.sh 0 _SYS_ snes %ROM%</command>
    <platform>satellaview,snes</platform> <!--Not used by Screenscraper, but by built-in ES scraper and maybe others-->
    <theme>satellaview</theme>
</system>
...
  • Line 3 defines the platform name, respective the folder name for your ROMs. Thus, Skyscraper expects to find ROMs in /home/pi/RetroPie/roms/satellaview.
  • Line 6 contains the extensions which are recognized by EmulationStation. These extensions should be also be present in the "formats": block of peas.json. However, Skyscraper uses case insensitive file extension mapping. The extensions .7z and .zip are added automagically by Skyscraper, thus the "formats": list is usually shorter than the EmulationStation <extension/> list.
  • Line 9: If your theme doesn't support Satellaview, you can also use snes as value.

Note

If you run a different frontend than EmulationStation, consult the documentation for your frontend on how to add additional systems.

Step 4: Happy Scraping

  1. Scrape and generate the satellaview/gamelist.xml as in the introductive use case using Skyscraper -p satellaview -s screenscraper and Skyscraper -p satellaview
  2. Restart EmulationStation, respective trigger reload of the gamelist in your frontend.
  3. Smile :)

Kudos

Thanks to retrobit @ GitHub for contributing this usecase.

Transferring Local Platform Changes

This section describes how to transfer your changes from peas.json and platforms_idmap.csv to *_local.* files with the same format. If you never changed these files, you can safely ignore this section.
Whenever there is an update and maybe changes to peas.json and platforms_idmap.csv Skyscraper will place the distribution files as peas.json.rp-dist and platforms_idmap.csv.rp-dist.
Before Skyscraper 3.15.0 you had to manually transfer updates from peas.json.rp-dist and platforms_idmap.csv.rp-dist. With Skyscraper 3.15.0 onwards there is a semi-automated approach.

Note

If you are using Skyscraper without RetroPie context these files will have the suffix .dist. The manual will use .rp-dist as synonym for both.

Step 1: Transfer Platform Information (peas) to Local File

Install Python Deepdiff: sudo apt install python3-deepdiff. Then navigate to /opt/retropie/supplementary/skyscraper/ and find the script deepdiff_peas_jsonfiles.py.

Tip

If you are using Skyscraper without RetroPie context the you can find the Python script in source in the supplementary/scraperdata folder.

The script expects at least two parameters:

  1. <source_peas.json>: The pristiine/baseline file with all platform information
  2. <dest_peas.json>: The file with your local changes
  3. Optionally <outfile_peas.json>: Once you have reviewed the changes provide this file to store the platform "diff" between the pristine file and your changes

Example(s)

Create a diff on the console: 

python3 deepdiff_peas_jsonfiles.py peas.json.rp-dist peas.json

Review the diff, then run: 

python3 deepdiff_peas_jsonfiles.py peas.json.rp-dist peas.json peas_local.json

Backup your peas.json if needed and when satisfied move peas.json.rp-dist to peas.json.

Step 2: Transfer Platform Scraper IDs (platforms_idmap) to Local File

The logic is the same as before and provides an output of the lines you have changed in platforms_idmap.csv in relation to the baseline platforms_idmap.csv.rp-dist.
Navigate to the folder with the platforms_idmap.csv. Then run:

diff \
  --new-line-format="%L" \
  --old-line-format="" \
  --unchanged-line-format="" \
  platforms_idmap.csv.rp-dist platforms_idmap.csv > platforms_idmap_local.csv

Also add the column header (folder, screenscraper_id, mobygames_id, tgdb_id) to the platforms_idmap_local.csv file, for example with:

echo folder,screenscraper_id,mobygames_id,tgdb_id | \
cat - platforms_idmap_local.csv > tmp_piggy.csv && \
mv tmp_piggy.csv platforms_idmap_local.csv

Backup your platforms_idmap.csv if needed and when satisfied move platforms_idmap.csv.rp-dist to platforms_idmap.csv.

Step 3: Automatically Apply Distribution Updates for Platform Configuration

Navigate to the folder with the _local.* files mentioned before. Create an empty file names .platformcfg_overwrite_ok (note the heading dot). If this file is present Skyscraper will overwrite the existing peas.json and platforms_idmap.csv with the files from the distribution when updating to a newer version.
If you remove the file .platformcfg_overwrite_ok, Skyscraper will install the *.rp-dist files again on the next install.

One More Thing...

If you have changes which would be beneficial for the community, feel free to file an issue with the proposed additions/changes or table it in the RetroPie Forum/Skyscraper Thread. Thank you!

Migrating platforms.json and screenscraper.json

Info

This section is only applicable if you update from Skyscraper 3.7.7-2.

Tip

If you neither edited platforms.json nor screenscraper.json or do not have these files in the Skyscraper config folder (sibling to the config.ini) you can safely ignore this section.

If you have local changes in these files you can transfer your local changes to the new files. The new peas.json is the platforms.json but in a more compact form which has less visual overhead and also allows faster parsing. The platforms_idmap.csv maps the former screenscraper.json platform ids plus the platform ids of two more web API sites.

Use the script convert_platforms_json.py (sibling to the Skyscraper executable) to convert the platforms.json to a peas_mine.json file which you can then diff to the peas.json and transfer your changes to peas_local.json.

Use the script check_screenscraper_json_to_idmap.py to identify differences from your screenscraper.json. Then use the three <scraper>_platforms.json files to identify the matching platform ids to be entered in platforms_idmap_local.csv. Use -1 in this file, if there is no matching platform id.

Summary of Changes in the Config Files

Filenames shown strikethrough are superseded.
Filenames shown italic are user editable.

File
 Introduced with Version Notes (version)
mobygames.json 3.8.0 superseded by mobygames_platforms.json (v3.9.0); not to be edited; IDs are used in platforms_idmap.csv
platforms.json 3.7.7-2 (@detain) superseded by peas.json (3.9.0) and peas_local.json (3.13.0); do edit the latter to add/change platforms; these files use a leaner format than the initially used platforms.json
platforms_idmap.csv 3.9.0 maps the platform names (handles) from peas.json / peas_local.json to exact platform IDs used in scrapers MobyGames, Screenscraper or TGDB; do edit to add new platforms
peas.json 3.9.0 maps platform names (read: ROM folder names) to extensions and aliases for that platform
peas_local.json 3.13.0 same as usage as peas.json, the _local.json file will not be altered by Skyscraper updates. Entries in this file have higher precedence than the distribution file peas.json
platforms_idmap_local.csv 3.15.0 same as usage as platforms_idmap.csv, the _local.csv file will not be altered by Skyscraper updates. Entries in this file have higher precedence than the distribution file platforms_idmap.csv
screenscraper.json 3.7.7-2 (@detain) IDs formerly used in here are part of platforms_idmap.csv (3.9.0); superseded by screenscraper_platforms.json (3.9.0) which is not to be edited
tgdb_developers.json 2.5.3 (@muldjord) API mapping of 'Developers'; Uses leaner format as before (3.9.0); not to be edited
tgdb_genres.json 3.9.0 API mapping of 'Genres' (3.9.0); not to be edited
tgdb_platforms.json 3.9.0 API mapping of 'Platforms' (3.9.0); not to be edited; IDs are used in platforms_idmap.csv
tgdb_publishers.json 2.5.3 (@muldjord) API mapping of 'Publishers'; Uses leaner format as before (3.9.0); not to be edited