Uploaded image for project: 'GPII - Global Public Inclusive Infrastructure'
  1. GPII - Global Public Inclusive Infrastructure
  2. GPII-1987

Implement data loaders in the universal repo to load CouchDB data

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Pull request
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      When setting up the GPII in the production mode, which is under development at this moment, the initial test data needs to be imported into CouchDB for the preferences server and the authorization server to lay a base ground for people to play with. Currently, this data loading process is performed by the docker image - https://github.com/gpii-ops/docker-preferences-server-data-loader

      More particularly, 2 ansible roles that are called up by the docker:

      https://github.com/gpii-ops/ansible-preferences-server-data-loader
      https://github.com/gpii-ops/ansible-preferences-server

      This means scripts written in ansible roles need to be aware of where the data, as well as the GPII production config file, are located in the GPII universal repo, its data structure, how to convert that data structure into CouchDB sensible structure etc. All these are hardcoded in ansible scripts so that any change on these information could break the data loading process. One existing breakage is https://issues.gpii.net/browse/GPII-1884, which is caused by the renaming of the GPII production config file.

      To solve this issue, data loaders need to be created and maintained in the GPII universal repo. These data loaders can be called/executed by external resources, the docker image in our case, to load data into CouchDB. This would help to isolate the universal specific info within the universal repo, reduce dependencies between the docker VM and the GPII universal, also reduce the amount of work required on the docker image.

      At the moment, 2 data loaders are needed:

      1. To convert and load prefs data into CouchDB for the preferences server;
      2. To load data and views into CouchDB for the authorization server.

      Email threads on this discussion:
      http://lists.gpii.net/pipermail/architecture/2016-August/004242.html
      http://lists.gpii.net/pipermail/architecture/2016-August/004243.html

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cli@ocad.ca Cindy Qi Li
                Reporter:
                cli@ocad.ca Cindy Qi Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: