How can I store 30.40 in DynamoDb?

[jonapich]

It's the end of the day, maybe i'm not seeing this clearly... But I am unable to store simple float values without adding some arcane magic to the mix.

Using .put_item on a Table resource that contains a float:

item = {'name': 'testing_row', 'foo': 30.40}
table.put_item(Item=item)
>> TypeError: Float types are not supported. Use Decimal types instead.

The same thing, using Decimal:

item = {'name': 'testing_row', 'foo': Decimal(30.40)}
table.put_item(Item=item)
>> Inexact: None

This last one should have gone through, no? The stack trace is this:

>       table.put_item(Item=item)

E:\Projects\Repos\tests\test_dynamodb.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Python27\lib\site-packages\boto3\resources\factory.py:518: in do_action
    response = action(self, *args, **kwargs)
C:\Python27\lib\site-packages\boto3\resources\action.py:83: in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
C:\Python27\lib\site-packages\botocore\client.py:258: in _api_call
    return self._make_api_call(operation_name, kwargs)
C:\Python27\lib\site-packages\botocore\client.py:524: in _make_api_call
    api_params, operation_model, context=request_context)
C:\Python27\lib\site-packages\botocore\client.py:574: in _convert_to_request_dict
    params=api_params, model=operation_model, context=context)
C:\Python27\lib\site-packages\botocore\hooks.py:227: in emit
    return self._emit(event_name, kwargs)
C:\Python27\lib\site-packages\botocore\hooks.py:210: in _emit
    response = handler(**kwargs)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:197: in inject_attribute_value_input
    'AttributeValue')
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:252: in transform
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:259: in _transform_parameters
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:274: in _transform_structure
    target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:259: in _transform_parameters
    model, params, transformation, target_shape)
C:\Python27\lib\site-packages\boto3\dynamodb\transform.py:283: in _transform_map
    params[key] = transformation(value)
C:\Python27\lib\site-packages\boto3\dynamodb\types.py:103: in serialize
    return {dynamodb_type: serializer(value)}
C:\Python27\lib\site-packages\boto3\dynamodb\types.py:204: in _serialize_n
    number = str(DYNAMODB_CONTEXT.create_decimal(value))
C:\Python27\lib\decimal.py:3938: in create_decimal
    return d._fix(self)
C:\Python27\lib\decimal.py:1712: in _fix
    context._raise_error(Inexact)

I can make it go through if I use the string trick in the Decimal constructor:

item = {'name': 'testing_row', 'foo': Decimal('30.40')}
table.put_item(Item=item)

Having done that, using AWS's dashboard to look into the table, I can see a number type with the value of 30.4 but then I cannot assert its value in my tests:

item = table.get_item(Key={'name': 'testing_row'})['Item']['foo']
assert item == 30.4
>> False

This seems to work:

assert float(item) == 30.4

So how's this supposed to work exactly? Is it expected that I must 1) provide a string to decimal and 2) convert back to float myself for equality to work properly?

[jonathanwcrane]

Yeah I've gotten this weird "inexact" error before. I think what I did was cast a number to decimal earlier on in the calculation, then I rounded it to two decimals points AFTER I had cast it, and it worked.

¯_(ツ)_/¯

[jonapich]

@jonathanwcrane but in my case i'm not doing any calculations. Boto3 forces me to convert my floats to Decimal, which I do, and then it is unable to push it... If you take a look at the stacktrace, you'll notice boto tries to do some funky stuff with my Decimal... this sounds like a bug.

[kyleknap]

@jonapich
You just need to enclose the float value with quotes. So something like this:

item = {'name': 'testing_row', 'foo': Decimal('30.40')}
table.put_item(Item=item)

Passing it as a string is the typical usage for Decimal in python: https://docs.python.org/2/library/decimal.html.

Let me know if that helps. I think an example should be added for this.

[jonathanwcrane]

So let's say it's a value that's calculated on the fly. Do we cast it as a string, thusly:

our_value = some_other_value / yet_another_value
item = {'name': 'testing_row', 'foo': Decimal(str(our_value))}
table.put_item(Item=item)

?

[jonapich]

@kyleknap did you look at the example I provided after the stack trace, in my original post?

I don't agree with this behavior being normal. Having to wrap these calls with home-made recursive conversion functions goes against the principle that you can usually just pass a boto resource around. It should be usable as-is.

I understand that python floats are annoying and unprecise, but boto really should handle the extra step of handling these conversions for the user. I agree that it shouldn't allow Inexact conversions, but for this it would be better to allow the user to attach his own conversion function when Inexact is raised.

[jonapich]

@jonathanwcrane the problem with this approach is that it also has to be converted back if you pull it:

>>> val1 = Decimal(30.40)
>>> val2 = Decimal('30.40')
>>> val1 == 30.40
True
>>> val2 == 30.40
False
>>> float(val1) == 30.40
True
>>> float(val2) == 30.40
True

[csimmons0]

Hi. I was hitting the same error and came up with this solution. If you're okay with rounding, you may find it helpful.

def round_float_to_decimal(float_value):
    """
    Convert a floating point value to a decimal that DynamoDB can store,
    and allow rounding.
    """

    # Perform the conversion using a copy of the decimal context that boto3
    # uses. Doing so causes this routine to preserve as much precision as
    # boto3 will allow.
    with decimal.localcontext(boto3.dynamodb.types.DYNAMODB_CONTEXT) as \
         decimalcontext:

        # Allow rounding.
        decimalcontext.traps[decimal.Inexact] = 0
        decimalcontext.traps[decimal.Rounded] = 0
        decimal_value = decimalcontext.create_decimal_from_float(float_value)
        g_logger.debug("float: {}, decimal: {}".format(float_value,
                                                       decimal_value))

        return decimal_value

[robolivable]

@jonapich +1, I totally agree with this. I shouldn't have to be yelled at for using floats in my values. And if they must be used, the library should be responsible for handling the conversions, not me.

What seems off here as well, is that this isn't an issue when using the DynamoDB.Client class approach for saving items.

[bittlingmayer]

We store a nested object, and don't want to write code that assumes a specific nesting or recursively searches for floats deep down in the tree. So for us the lazy workaround is to use json.dump and store the entire dict as string.

[Alonreznik]

I've solved that issue by creating a json.loads object hook, which can make that conversion from decimal to float.

from dateutils import parser
from boto3.dynamodb.types import TypeDeserializer, TypeSerializer

def encode_object_hook(dct):
    try:
        return parser.parse(TypeDeserializer().deserialize(dct))
    except (ValueError, AttributeError, TypeError):
        try:
            val = TypeDeserializer().deserialize(dct)
            if isinstance(val, Decimal):
                if val % 1 > 0:
                    return float(val)
                elif val < maxint:
                    return int(val)
                else:
                    return long(val)
            else:
                return val
        except:
            return dct

For "dumps" your dict into DynamoDB put_item valid value, you can use this serializer:

def decode_object_hook(dct):
    for key, val in dct.iteritems():
        if isinstance(val, float):
            dct[key] = Decimal(str(val))
        try:
            dct[key] = TypeSerializer().serialize(val)
        except:
            dct[key] = val
    return dct

def json_serial(val):
    if isinstance(val, datetime):
        serial = val.strftime('%Y-%m-%dT%H:%M:%S.%f')
        return serial
    elif isinstance(val, set):
        serial = list(val)
        return serial
    elif isinstance(val, uuid.UUID):
        serial = str(val.hex)
        return serial

def dumps(dct, *args, **kwargs):
    kwargs['object_hook'] = decode_object_hook
    return json.loads(json.dumps(dct, default=json_serial), *args, **kwargs)

item = dumps({'name': 'testing_row', 'foo': 30.40})
table.put_item(Item=item)