Usage

This section explains Satellite Populate data populate.

Commands

Using $ satellite-populate you can run the populate and validate commands. That commands are used to read data description from YAML file and populate the system or validate populated entities.

Having test_data.yaml with the following content.

vars:
  org_label_suffix = inc
actions:
  - model: Organization
    log: The first organization...
    register: org_1
    data:
      name: MyOrg
      label: MyOrg{{org_label_suffix}}

To populate the system

(satellite_env)[you@host]$ satellite-populate test_data.yaml -v -o validation_data.yaml
2017-01-04 04:31:17 - satellite_populate.base - INFO - CREATE: The first organization...
2017-01-04 04:31:19 - satellite_populate.base - INFO - search: Organization {'query': {'search': 'name=MyOrg,label=MyOrg'}} found unique item
2017-01-04 04:31:19 - satellite_populate.base - INFO - create: Entity already exists: Organization 36
2017-01-04 04:31:19 - satellite_populate.base - INFO - registry: org_1 registered

To validate the system use the file generated by population validation_data.yaml

(satellite_env)[you@host]$ satellite-populate validation_data.yaml
(satellite_env)[you@host]$ echo $?
0  # system validated else 1

Use $ satellite-populate --help for more info

Hostname and Credentials

Pass -h –hostname, -p –password, -u –username to the command, or this arguments to decorator:

@populate_with(data, username='x', password='y', hostname='server.com')

NOTE:

“validation data can also be included in config section”

Decorator

Other way to use populate is via decorator, it is useful to decorate a test_case forcing a populate or validate operation to be performed.

Having a data_file like:

actions:
  - model: Organization
    register: organization_1
    data:
      name: My Org

Then you can use in decorators:

@populate_with('file.yaml')
def test_case_(self):
    'My Org exists in system test anything here'

And getting the populated entities inside the test_case:

@populate_with('file.yaml', context_name='my_context')
def test_case_(self, my_context=None):
    assert my_context.organization_1.name == 'My Org'

You can also set a customized context wrapper to the
context_wrapper argument::

    def my_custom_context_wrapper(result):
        # create an object using result
        my_context = MyResultContext(result)
        return my_context

    @populate_with('file.yaml', context_name='my_context',
                   content_wrapper=my_custom_context_wrapper)
    def test_case_(self, my_context=None):
        # assert with some expression using my_context object returned
        # my_custom_context_wrapper
        assert some_expression

And if you don’t want to have YAML file you can provide a dict:

data_in_dict = {
    'actions': [
        {
            'model': 'Organization',
            'register': 'organization_1',
            'data': {
                'name': 'My Organization 1',
                'label': 'my_organization_1'
            }
        },
    ]
}


@populate_with(data_in_dict, context_name='my_context', verbose=1)
def test_org_1(my_context=None):
    """a test with populated data"""
    assert my_context.organization_1.name == "MyOrganization1"

And finally it also accepts bare YAML string for testing purposes:

data_in_string = """
actions:
- model: Organization
  registry: organization_3
  data:
    name: My Organization 3
    label: my_organization_3
"""

@populate_with(data_in_string, context_name='context', verbose=1)
def test_org_3(context=None):
    """a test with populated data"""
    assert context.organization_3.name == "My Organization 3"
    assert context.organization_3.label == "my_organization_3"

NOTE:

“That is important that context_name argument always be declared using either a default value my_context=None or handle in **kwargs Otherwise py.test may try to use this as a fixture placeholder. And if context_wrapper is set to None, my_context will be the pure unmodified result of populate function.”

Decorating UnitTest setUp and test_cases:

class MyTestCase(TestCase):
    """
    This test populates data in setUp and also in individual tests
    """
    @populate_with(data_in_string, context_name='context')
    def setUp(self, context=None):
        self.context = context

    def test_with_setup_data(self):
        self.assertEqual(
            self.context.organization_3.name, "My Organization 3"
        )

    @populate_with(data_in_dict, context_name='test_context')
    def test_with_isolated_data(self, test_context=None):
        self.assertEqual(
            test_context.organization_1.name, "My Organization 1"
        )

The YAML data file

In the YAML data file it is possible to specify 3 sections, config, vars and actions.

config

The config may be used to define special behavior of populator and its keys are:

example:

config:
  verbose: 3
  populator: api
  populators:
    api:
      module: satellite_populate.api.APIPopulator
    cli:
      module: satellite_populate.cli.CLIPopulator

Config variables:

config:
  # Set verbosity to -v, -vv, -vvv, -vvvv, -vvvvv
  # int
  # range(0, 5)
  verbose: 1

  # define the default active populator name
  # str
  populator: foo

  # specify available populators
  # dict(<name>=dict(module='module_path'))
  populators:
    foo:
      module: mypack.mymodule.MyPopulatorClass
    other:
      module: otherpath.OtherClass

  # define the mode (override by argument)
  # str
  # choices: validate | populate
  mode: validate

  # http or https ? (override by argument)
  schema: http

  # Satellite system port (override by argument)
  port: 443

  # hostname (without scheme) (override by argument)
  hostname: server.com

  # Admin username (override by argument)
  username: admin

  # admin password (override by argument)
  password: changeme

  # User for ssh login (override by argument)
  ssh_user: root

  # Ssh auth (override by argument)
  # if None local ~/.ssh pub key is used
  # or password
  # or keyfile
  ssh_auth:
    password: 123456
    key_file: path/to/file.pub

  # raw search rules is a dict of rules
  # to force some transformations over nailgun
  # EntitySearchMixin
  # in the example below we are removing the password
  # field from search queries for User entity
  raw_search_rules:
    user:
      password:
        remove: true

  # In some cases a GPGKey is needed for nailgun
  gpgkey:
    content: skjfsdhbgbsdhbgsdjbg=
    docker_url: system.com:dockerport

  # inject following modules to context (import)
  add_to_context:
    path: os.path
    shortname: package.module.module.module.object
    # the above will available as {{ shortname }}

vars

Variables to be available in the rendering context of the YAML data every var defined here is available to be referenced using Jinja syntax in any action.

vars:
  admin_username: admin
  admin_password: changeme
  org_name_list:
    - company7
    - company8
  prefix: aaaa
  suffix: bbbb
  my_name: me

Actions

The actions is the most important section of the YAML, it is a list of actions being each action a dictionary containing special keys depending on the action type.

Actions are executed in the defined order and order is very important because each action can register its result to the internal registry to be referenced later in any other action.

The action type is defined in action key and available actions are:

CRUD

Crud actions takes a model argument, any from nailgun.entities is a valid model, models are passed as CamelCasedName of the antity class, then, depending on the populator being used, that CRUD action can be performed by API, CLI or UI.

List of possible variables for crud actions:

# action name - create | delete | update
action: create

# entity class
model: User

# name to register
register: my_user

# log message to output
log: Creating a new user ....

# Must iterate a list to repeat the same action?
with_items:
  - item1
  - item2
  ...

# The data to perform a search for the entity
data:

  # base types - int, str, list etc..
  name: Foo bar

  # from an available Python object
  url:
    from_object: somemodule.constants.REPO_URL

  # from a search in the system
  organization:
    from_search:
      model: Organization
      data:
        name: SomeCompanyName

  # from specific ID
  product:
    from_read:
      model: Product
      data:
        id: 1

  # from registered action
  user:
    from_registry: already_existing_user

  # from fauxfactory generator
  password:
    from_factory: alphanumeric

# If needed specify data to be used only for search (in validation)
search_query:
  field: something

# If needed custom options can be passed to nailgun search
search_options:
  filter: {}

  # should force a raw search or use attribute search?
  # note: some entities such as Organization will always be raw searched
  force_raw: true | false

# Choose which populator to use for this specific action
# NotImplementedYet
via: api | cli | ui | custom_populator

# Should errors be silenced and None registered if error?
silent_errors: true | false

# Run async?
# NotImplementedYet
async: true | false
wait: other_action_register_name

# Run only in the case of following condition
# Python allowed, registered objects allowed
# should be a Boolean operation
when: object_a == object_b and 1 > 0
create

Search for the new entity and creates if not found, else only register the object.

  • If no action is informed create will be always the default
  • In populate perform search then create
  • In validate perform only search

Required variables:

  • model: Nailgun Entity Class name
  • data: a dictionary to search or populate the entity

Creating a simple Organization:

# a list of dictionaries
actions:

  - model: Organization  # the nailgun Entity class

    # The message to output in the log
    log: This is the first organization

    # The name which this object will be registered
    # to be referenced in other actions.
    register: my_organization

    # The data to search or populate the entity
    data:
      name: My Company
      label: mycompany

Creating 2 organizations and 2 users from lists and referencing objects from the registry:

vars:

  # a list with data for 2 users
  user_list:
    - firstname: Michael
      lastname: Scott
    - firstname: David
      lastname: Brent

  # a list of company names
  company_names:
    - Dunder Mifflin
    - Wernham Hogg

actions:

  # create all the organizations listed above
  - model: Organization

    # iterate specified list and repeats the action for each
    with_items: company_names

    # include the result in registry
    # if `with_items` is used, the registered object will be a list
    register: companies

    # give the data
    data:
      name: "{{item}}"
      label: "{{item.replace(' ', '')}}"  # transform name in a valid label

  # Create one user as admin for each organization
  - model: User
    with_items: user_list
    data:
      admin: true
      # refer to loop iteration using `items` object
      firstname: "{{item.firstname}}"
      lastname: "{{item.lastname}}"

      # Use object methods and Jinja filters to transform data
      # the following gives us mscott and dbrent
      login: "{{ '{0}{1}'.format(item.firstname[0], item.lastname) | lower }}"

      # generate a random password using builtin fauxfactory
      password:
        from_factory: alpha

      # Set the organizations to existing list of orgs
      organization:
        from_registry: companies

      # Set as default org the same positioned in the loop
      default_organization:
        from_registry: companies[loop_index]
update

Get some existing entity and updates it with provided data.

  • Executed only in populate mode
  • In validate mode it only searches for updated entity

Required variables:

  • model: Nailgun Entity Class name
  • registry The name registry object
  • data: a dictionary to search

Updating the product named old_name with new_name:

actions:
   - action: update
     model: Product
     register: some_product
     data:
       name: new_name
     search_query:
       name: old_name
       organization:
           from_search:
             model: Organization
             data:
               name: Default Organization

If the some_product already exists in registry you can omit the search:

actions:
   - action: update
     model: Product
     register: some_product
     data:
       name: new_name
delete

Deletes existing entity.

  • Executed only in populate mode
  • In validate mode it only searches for updated entity

Required variables:

  • model: Nailgun Entity Class name
  • registry The name registry object
  • data: a dictionary to search

Deleting the product named new_name:

actions:
   - action: delete
     model: Product
     search_query:
       name: new_name
       organization:
           from_search:
             model: Organization
             data:
               name: Default Organization

If the some_product already exists in registry you can omit the search:

actions:
   - action: delete
     model: Product
     register: some_product

Note:

“delete action perform a DELETE call to the api and removes the entity from the system, while unregister action only removes it from runtime registry”

OTHER

This are other built-in actions

echo

Outputs a message to the LOG and also to stdout.

Required variables:

  • log: The message to be logged

Examples:

actions:
   - action: echo
     log: Hello World
   - action: echo
     log: This an error
     level: error
   - action echo
     log: This message goes also to the stdout
     print: true
   - action: echo
     log: I can read variables, you are {{ env.USER }}

Which outputs:

2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: Hello World
2017-01-20 00:10:53 - satellite_populate.base - ERROR - ECHO: This an error
2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: This message goes also to the stdout
This message goes also to the stdout
2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: I can read variables, you are root
register

Register variables to the runtime registry

Required variables:

  • data: A dictionary

Examples:

- action: register
  data:
    name: Michael Scott
    preferred_organization:
      from_search:
        model: Organization
        data:
          name: My prefered Organization
    repo_url:
      from_object: "http://" + file.constants.REPO_BASE_URL

All variables registered above will be available for the next executed actions.

unregister

Removes variables from runtime register.

Required variables:

  • data: A list of variable names

Examples:

- action: unregister
  data:
    - name
    - preferred_organization
    - repo_url

All variables unregistered above will be not available for the next executed actions.

Unregister is useful for actions using when: conditions.

assertion

Execute predefined assertion operations and fails the validation if assertion returns False.

Required variables:

  • operator: Logical operator mapped to a function returning Boolean
  • data: A list of two elements to be tested

Built in operators:

  • eq # the default
  • ne
  • gt
  • lt
  • gte
  • lte
  • identity

Examples:

- action: assertion
  log: Check if current user is root
  operator: eq
  data:
    - root
    - "{{ env.USER }}"

If returns False, the validation ends with exit code 1

Custom Populators can also include custom operators for assertion.

CUSTOM

And you can also have special actions defined in a custom populator.

Lets say you have this python module in your project, properly available on PYTHONPATH:

from satellite_populate.api import APIPopulator

class MyPopulator(APIPopulator):
    def action_writeinfile(self, rendered_data, action_data):
        with open(rendered_data['path'], 'w') as output:
            output.write(rendered_data['content'])

Now go to your test.yaml and write:

config:
  populator: mine
  populators:
    mine:
      module: mypath.mymodule.MyPopulator

actions:

  - action: writeinfile
    path: /tmp/test.txt
    content: Hello World!!!

and run:

$ satellite-populate test.yaml -v

Dynamic Data

There are some ways to fetch dynamic data in action definitions, it depends on the action type.

For any key you can use Jinja to provide a dynamic value as in:

value: "{{ get_something }}"
value: "{{ fauxfactory.gen_string('alpha') }}"
value: user_{{ item }}

For some actions you can provide a data key, that data is used to create new entities and also to perform searches or build the action function.

Every data key accepts 4 special reference directives in its sub-keys.

  • from_registry

    Gets anything from registry:

    data:
      organization:
        from_registry: default_org
      name:
        from_registry: my_name
    
  • from_object

    Gets any Python object available in the environment:

    data:
      url:
        from_object:
          name: robottelo.constants.FAKE_0_YUM_REPO
    
  • from_search

    Perform a search and return its result:

    data:
      organization:
        from_search:
          model: Organization
          data:
            name: Default Organization
    
  • from_read

    Perform a read operation, which is useful when we have unique data or id:

    data:
      organization:
        from_read:
          model: Organization
          data:
            id: 1
    

The internal registry

Every action which returns a result can write its result to the registry, so it is available to be accessed by other actions.

Provide a register unique name in action definition.

The actions that support register are:

  • create
  • update
  • register
  • assertion

All dynamic directives from_* supports the use of register

Example:

- action: create
  model: Organization
  register: my_org
  data:
    name: my_org

- model: User
  log: Creating user under {{ register.my_org.name }}
  data:
    organization:
      from_registry: my_org