How can I convert a dict object(with Map Type) to a thrift object?

[liuweiming1997]

HI guys.
I am facing a bug that can`t parse python dict to the thrift object. Here is the simple.
Can anyone help me? Thanks a lot.

enum.thrift

namespace c_glib base
namespace cpp base
namespace py base
namespace go base
namespace java base
namespace js base

enum Status {
	SUCCESS = 1,
	FAIL = 2,
}

struct Children {
  1: optional string name,
}

struct Person {
  1: optional string id,
  2: optional string url,
  3: optional string appid,
  4: optional i64 submit_time,
  5: optional Status person_status,
  6: optional Children children,

  10: optional map<string, string> params,
}

service post_data {
  Status do_post(1: Person person)
}

main.py

import thriftpy2

from thriftpy2.rpc import client_context
import thriftpy2.protocol.json as thrift_json_tool

enum_thrift = thriftpy2.load("enum.thrift", module_name="enum_thrift")

person = enum_thrift.Person()

person_dict = {
    'id': '1',
    'url': 'http://www.google.com',
    # this filed can not parse
    'params': {
        'xxx': 'bbb',
        'yyy': 'ccc',
    }
}

thrift_json_tool.struct_to_obj(val=person_dict, obj=person)

print(person)

Error:

Traceback (most recent call last):
  File "thrift_py2_use.py", line 20, in <module>
    thrift_json_tool.struct_to_obj(val=person_dict, obj=person)
  File "/home/weimingliu/.local/lib/python3.5/site-packages/thriftpy2/protocol/json.py", line 149, in struct_to_obj
    obj_value(field_type, val[field_name], field_type_spec))
  File "/home/weimingliu/.local/lib/python3.5/site-packages/thriftpy2/protocol/json.py", line 59, in obj_value
    return func(*args)
  File "/home/weimingliu/.local/lib/python3.5/site-packages/thriftpy2/protocol/json.py", line 75, in map_to_obj
    value_type, v["value"], value_spec)
TypeError: string indices must be integers

[liuweiming1997]

DO we need this format?

{'params': [{'key': 'c', 'value': 'd'}, {'key': 'a', 'value': 'b'}]}

[JonnoFTW]

That particular function is intended to convert dicts in the thriftpy2 json format into thrift objects. If you have a simple dict and want to make it into a thrift object. You can always do:

person = enum_thrift.Person(**person_dict)

This only works for simple thrift structs (ie. ones that aren't nested). Otherwise you'll have to write something recursive that reads the thrift spec. It might look like:

def dict_to_thrift(thrift_cls, data, ignore_missing=True):
    """
    Convert a dict to a thrift object

    :param thrift_cls: the thrift class
    :param data: the data to be encoded
    :param ignore_missing: fail if we are missing a value
    :return:
    """
    result = {}
    if isinstance(data, (str, int, float, bool, bytes)):
        return data
    if isinstance(thrift_cls, tuple):
        container_type = thrift_cls[0]
        item_type = thrift_cls[1]
        if container_type == TType.STRUCT:
            return dict_to_thrift(item_type, data, ignore_missing)
        elif container_type in (TType.LIST, TType.SET):
            return [dict_to_thrift(item_type, v, ignore_missing) for v in data]
        elif container_type == TType.MAP:
            return {
                dict_to_thrift(item_type[0],k, ignore_missing):
                    dict_to_thrift(item_type[1], v, ignore_missing) for k, v in data.items()
            }
    for field_idx, spec in thrift_cls.thrift_spec.items():
        thrift_type, field_name = spec[0], spec[1]
        if field_name not in data:
            if ignore_missing:
                continue
            else:
                raise ValueError(f"Missing non-optional field {field_name}")
        dict_data = data[field_name]
        # handle each type here
        if thrift_type in (TType.LIST, TType.SET):
            result[field_name] = [dict_to_thrift(spec[2], x, ignore_missing) for x in dict_data]
        elif thrift_type == TType.STRUCT:
            result[field_name] = dict_to_thrift(spec[2], dict_data, ignore_missing)
        elif thrift_type == TType.MAP:
            result[field_name] = {
                dict_to_thrift(spec[2][0], k, ignore_missing):
                    dict_to_thrift(spec[2][1], v, ignore_missing) for k, v in
                dict_data.items()}
        else:
            result[field_name] = dict_data
    if hasattr(thrift_cls, '__call__'):
        return thrift_cls(**result)
    else:
        for k, v in result.items():
            setattr(thrift_cls, k, v)
        return thrift_cls

This is adapted from apache_json.py

Note that there's no way to tell if a field is marked as optional from the .thrift_spec, so use at your own risk.

[ms300]

@JonnoFTW I have the same issue and thanks for the dict_to_thrift function.
And there is a little problem in the code.

        # handle each type here
        if thrift_type in (TType.LIST, TType.SET):
            result[field_name] = [dict_to_thrift(spec[2], x, ignore_missing) for x in dict_data]
        if thrift_type == TType.STRUCT:
            result[field_name] = dict_to_thrift(spec[2], dict_data, ignore_missing)
        if thrift_type == TType.MAP:
            result[field_name] = {
                dict_to_thrift(spec[2][0], k, ignore_missing):
                    dict_to_thrift(spec[2][1], v, ignore_missing) for k, v in
                dict_data.items()}
        else:
            result[field_name] = dict_data

should be

        # handle each type here
        if thrift_type in (TType.LIST, TType.SET):
            result[field_name] = [dict_to_thrift(spec[2], x, ignore_missing) for x in dict_data]
        elif thrift_type == TType.STRUCT:
            result[field_name] = dict_to_thrift(spec[2], dict_data, ignore_missing)
        elif thrift_type == TType.MAP:
            result[field_name] = {
                dict_to_thrift(spec[2][0], k, ignore_missing):
                    dict_to_thrift(spec[2][1], v, ignore_missing) for k, v in
                dict_data.items()}
        else:
            result[field_name] = dict_data

[JonnoFTW]

@ms300 thanks, I've updated my code

[liuweiming1997]

Thanks all guy`s useful code.
I suggest this function can maintain by the repos.

[JonnoFTW]

@liuweiming1997 please keep in mind that thriftpy2 doesn't currently support optional fields since this information is excluded from the thrift_spec attribute of thrift objects.