serialize non primitive types

[PanagiotisDrakatos]

I use this library in my project and works very well with strings, integers,bytes, etc and i am very happy but when i tried to use object classes from dependencies, the serialization did not work well.

Assuming this library and a simple example

 <dependency>
      <groupId>org.miracl.milagro.amcl</groupId>
      <artifactId>milagro-crypto-java</artifactId>
      <version>0.4.0</version>
    </dependency>

import io.activej.serializer.annotations.Deserialize;
import io.activej.serializer.annotations.Serialize;
import io.activej.serializer.annotations.Deserialize;
import io.activej.serializer.annotations.Serialize;
import org.apache.milagro.amcl.BLS381.ECP;



public class Br {
    private ECP str;

    public Br(@Deserialize("str") ECP str) {
        this.str = str;
    }

    @Serialize
    public ECP getStr() {
        return str;
    }

    public void setStr(ECP str) {
        this.str = str;
    }

    @Override
    public String toString() {
        return "Br{" +
                "str='" + str + '\'' +
                '}';
    }
}

public class Ar {

    private String sar;
    private Br val;

    public Ar(@Deserialize("sar") String sar, @Deserialize("val") Br val) {
        this.sar = sar;
        this.val = val;

    }

    @Serialize
    public String getSar() {
        return sar;
    }

    @Serialize
    public Br getVal() {
        return val;
    }

    public void setVal(Br val) {
        this.val = val;
    }

    public void setSar(String sar) {
        this.sar = sar;
    }


    @Override
    public String toString() {
        return "Ar{" +
                "sar='" + sar + '\'' +
                ", val=" + val +
                '}';
    }
}

import io.activej.serializer.BinarySerializer;
import io.activej.serializer.SerializerBuilder;
import org.apache.milagro.amcl.BLS381.ECP;
import org.spongycastle.util.encoders.Hex;

public class AppSer {
    public static void main(String[] args) {
        byte[] buffer = new byte[200];
        BinarySerializer<Ar> enc = SerializerBuilder.create().build(Ar.class);
        Br s=new Br(ECP.fromBytes(Hex.decode("021342a49692039a716c41284ade085c092d3e966085ace49251f34da2a9941e517758ca9630f4f4e80a84cf6bcfae9708000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000")));
        Ar ar=new Ar("test",s);


        System.out.println(ar.toString());
        enc.encode(buffer, 0,ar);
        Ar copy =  enc.decode(buffer, 0);
        System.out.println(copy.toString());

    }
}

The output is this:

Ar{sar='test', val=Br{str='(1342a49692039a716c41284ade085c092d3e966085ace49251f34da2a9941e517758ca9630f4f4e80a84cf6bcfae9708,05ba13b1906193da8c22eeac7772c8e9dcf2ffca66b3c7081a9deee80dfc29bce838bca4a33c4f7516d658e9bd456e80)'}}
Ar{sar='test', val=Br{str='infinity'}}

but the two comparable values are not equal, also the data seems lost during serialization why this has happened?

In order to solve my problem I was trying to write custom serializers but with no success but I am barely sure that I am close this is my custom serializer class:

import io.activej.codegen.expression.Expression;
import io.activej.codegen.expression.Variable;
import io.activej.serializer.AbstractSerializerDef;
import io.activej.serializer.CompatibilityLevel;
import org.apache.milagro.amcl.BLS381.ECP;

import static io.activej.codegen.expression.Expressions.*;
import static io.activej.serializer.impl.SerializerExpressions.*;

public class CustomSerializer extends AbstractSerializerDef {

    @Override
    public Class<?> getEncodeType() {
        return ECP.class;
    }

    @Override
    public Expression encoder(final StaticEncoders staticEncoders,
                              final Expression buf,
                              final Variable pos,
                              final Expression ecp,
                              final int version,
                              final CompatibilityLevel compatibilityLevel) {
        //return sequence(writeBytes(buf, pos, call(ecp, "toBytes",value(new byte[2048]),value(true))));
        return let(call(ecp, "toBytes",value(new byte[2048]),value(true)), bytes -> sequence(
                writeVarInt(buf, pos, length(bytes)),
                writeBytes(buf, pos, bytes)));
    }

    @Override
    public Expression decoder(final StaticDecoders staticDecoders,
                              final Expression in,
                              final int version,
                              final CompatibilityLevel compatibilityLevel) {
      //  return staticCall(ECP.class, "fromBytes", readBytes(input,input));
        return let(arrayNew(byte[].class, readVarInt(in)), array ->
                staticCall(ECP.class, "fromBytes", readBytes(in,array)));
    }
}

i am trying to seriliaze tobytes method and decode frombytes. this is the source code to take look at this method.

BinarySerializer<Ar> enc = SerializerBuilder.create().with(ECP.class, ctx -> new CustomSerializer()).build(Ar.class);
The exception that i got is this:

Exception in thread "main" java.lang.IllegalArgumentException: Static method not found: java.lang.System.arraycopy(void,int,[B,int,int)
	at io.activej.codegen.Context.invokeStatic(Context.java:333)
	at io.activej.codegen.Context.invokeStatic(Context.java:312)
	at io.activej.codegen.expression.ExpressionStaticCall.load(ExpressionStaticCall.java:40)
	at io.activej.codegen.expression.ExpressionSequence.load(ExpressionSequence.java:44)
	at io.activej.codegen.expression.ExpressionSequence.load(ExpressionSequence.java:44)
	at io.activej.codegen.expression.ExpressionSequence.load(ExpressionSequence.java:44)
	at io.activej.codegen.expression.ExpressionSequence.load(ExpressionSequence.java:44)
	at io.activej.codegen.ClassBuilder.toBytecode(ClassBuilder.java:473)
	at io.activej.codegen.ClassBuilder.toBytecode(ClassBuilder.java:385)
	at io.activej.codegen.ClassBuilder.toBytecode(ClassBuilder.java:374)
	at io.activej.codegen.DefiningClassLoader.lambda$ensureClass$1(DefiningClassLoader.java:151)
	at io.activej.codegen.DefiningClassLoader.ensureClass(DefiningClassLoader.java:236)
	at io.activej.codegen.DefiningClassLoader.ensureClass(DefiningClassLoader.java:151)
	at io.activej.codegen.DefiningClassLoader.ensureClassAndCreateInstance(DefiningClassLoader.java:167)
	at io.activej.serializer.SerializerBuilder.build(SerializerBuilder.java:466)
	at io.activej.serializer.SerializerBuilder.build(SerializerBuilder.java:446)
	at Seriliaze.dsa.AppSer.main(AppSer.java:11)

I think the problem of the exception is that i call toBytes method which is void, i must somehow to get the first parameter of void as an expression something like this, i am not sure, how to deal with it?

[eduard-vasinskyi]

Hi, @PanagiotisDrakatos

You got the right idea that you need a custom serializer for the ECP class. There are a few comments though. I looked at the source code of the ECP#toBytes method and it looks like it doesn’t use more than BIG.MODBYTES + 1 bytes of the target array. So you can save some space by creating an array of that size instead of 2048. You also don’t have to encode the length of the array, since it always is BIG.MODBYTES + 1. Also, when decoding you need to pass the array to the fromBytes static factory, while readBytes() returns an int.

Here is the updated CustomSerializer class:

public class CustomSerializer extends AbstractSerializerDef {
 
  private static final Expression ARRAY_LEN = value(BIG.MODBYTES + 1);
 
  @Override
  public Class<?> getEncodeType() {
     return ECP.class;
  }
 
  @Override
  public Expression encoder(final StaticEncoders staticEncoders,
        final Expression buf,
        final Variable pos,
        final Expression ecp,
        final int version,
        final CompatibilityLevel compatibilityLevel) {
     return let(arrayNew(byte[].class, ARRAY_LEN),
           array -> sequence(
                 call(ecp, "toBytes", array, value(true)),
                 writeBytes(buf, pos, array)));
  }
 
  @Override
  public Expression decoder(final StaticDecoders staticDecoders,
        final Expression in,
        final int version,
        final CompatibilityLevel compatibilityLevel) {
     return let(arrayNew(byte[].class, ARRAY_LEN),
           array -> sequence(
                 readBytes(in, array),
                 staticCall(ECP.class, "fromBytes", array)));
  }
}

BTW, when writing your own SerializerDef it is sometimes useful to create a SerializerBuilder with a DefiningClassLoader that has a debug output directory configured.

SerializerBuilder.create(
    DefiningClassLoader.create()
        .withDebugOutputDir(debugPath)
)

That way you can inspect the debug folder to see the generated classes.

[PanagiotisDrakatos]

@eduard-vasinskyi Thank you very much for your wonderful answer it saves my day. i really appreciate it. You did an awesome job with activeJ i really admire your work and i already use a lot of things as libraries in my project. wish you the best, my friend have a great day

[eduard-vasinskyi]

@PanagiotisDrakatos

BTW, we have added an abstract class SimpleSerializerDef that you can extend in order to simplify the creation of serializers for custom classes. This class allows you to define BinarySerializer directly without needing Expressions API.

With SimpleSerializerDef, your serializer for the ECP class would look like this:

public class CustomSerializer extends SimpleSerializerDef<ECP> {
 
  @Override
  protected BinarySerializer<ECP> createSerializer(int version, CompatibilityLevel compatibilityLevel) {
     return new BinarySerializer<ECP>() {
        @Override
        public void encode(BinaryOutput out, ECP item) {
           byte[] bytes = new byte[BIG.MODBYTES + 1];
           item.toBytes(bytes, true);
           out.write(bytes);
        }
 
        @Override
        public ECP decode(BinaryInput in) throws CorruptedDataException {
           byte[] bytes = new byte[BIG.MODBYTES + 1];
           in.read(bytes);
           return ECP.fromBytes(bytes);
        }
     };
  }
}

This change is part of ActiveJ v5.4.3, which was released today.

[PanagiotisDrakatos]

@eduard-vasinskyi
Awesome!!! you did great work i am really impressed and hyped with ActiveJ. One fast quick question assuming we have a class that we want to serialize that has some huge objects e.g array lists hashmaps is there any way to dynamically calculate the size of the buffer that it will be used for serialization? Thus to avoid static initialize like this:

          byte[] buffer = new byte[200];
          BinarySerializer<HugeClass> enc = SerializerBuilder.create().build(HugeClass.class);
          enc.encode(buffer, 0,a);

[eduard-vasinskyi]

@PanagiotisDrakatos

There is no built-in way to pre-calculate the size of the buffer. You could add an external method for estimating a buffer size individually for the serialized object (if you can predict the size beforehand).

Another approach that we use for serializing a stream of data is to start from the small buffer size and then increase it if we underestimated the size of the serialized object. A BinarySerializer will throw an ArrayIndexOutOfBoundsException if there is not enough space in the buffer.

[PanagiotisDrakatos]

@eduard-vasinskyi ok thnx for your time i will search it further. Your approach is not bad i will consider implement it