Removals from an ArrayCollection gets normalized wrong

[Drachenkaetzchen]

Hi,

I just noticed that if one removes an element from a collection, the result is wrong.

Example

/**
 * @ORM\Entity
 */
class Project
{
    /**
     * @ORM\OneToMany(targetEntity="PartKeepr\ProjectBundle\Entity\ProjectPart",mappedBy="project",cascade={"persist", "remove"})
     * @Groups({"default"})
     * @var ArrayCollection
     */
    private $parts;

    public function __construct () {
        $this->parts = new ArrayCollection();
    }

    /**
     * Returns the parts array
     *
     * @return ProjectPart[] An array of ProjectPart objects
     */
    public function getParts()
    {
        return $this->parts;
    }

     /**
     * Adds a Project Part
     *
     * @param ProjectPart $projectPart An attachment to add
     */
    public function addPart($projectPart)
    {
        $projectPart->setProject($this);
        $this->parts->add($projectPart);
    }

    /**
     * Removes a Project Part
     *
     * @param ProjectPart $projectPart An attachment to remove
     */
    public function removePart($projectPart)
    {
        $projectPart->setProject(null);
        $this->parts->removeElement($projectPart);
    }
}

Note that the example class is barebones and doesn't include e.g. an identifier, which is actually the case in my project.

When I remove any part except the first, the resulting JSON has array indexes in the result:

{  
   "@context":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/contexts\/Project",
   "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/projects\/25",
   "@type":"Project",
   "parts":{  
      "0":{  # This index is included which is otherwise missing
         "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/project_parts\/758",
         "@type":"ProjectPart",
         "part":{  
            "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/parts\/284",
            "@type":"Part",
       }
}

I would expect the resulting JSON like this:

{  
   "@context":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/contexts\/Project",
   "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/projects\/25",
   "@type":"Project",
   "parts":{  
      {  # No index
         "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/project_parts\/758",
         "@type":"ProjectPart",
         "part":{  
            "@id":"\/~felicitus\/PartKeepr\/web\/app_dev.php\/api\/parts\/284",
            "@type":"Part",
       }
}

If my adder/remover implementation is wrong, please let me know. If so, I must refer to #153 (comment) again (Sample Implementation of getters/adders/removers)

[Drachenkaetzchen]

I think the root cause for this is with the json_encode function of PHP:

$data = array(
        0 => array("foo" => "bar"),
        1 => array("foo" => "bar")
);

echo json_encode($data);

$data2 = array(
        2 => array("foo" => "bar"),
        1 => array("foo" => "bar")
);

echo json_encode($data2);

Results in:

[{"foo":"bar"},{"foo":"bar"}]
{"2":{"foo":"bar"},"1":{"foo":"bar"}}

I wonder what the correct workaround is. sort() unfortunately doesn't work on ArrayCollection, and there's no option one could pass to json_encode() to ignore .

I tried the following approach, which works:

    public function getParts()
    {
        return array_values($this->parts->getIterator()->getArrayCopy());
    }

However, this is counter-intuitive and would have to be documented. I'm also not certain about the performance…

[theofidry]

@FELICITUS I've already encountered the problem and unfortunately there is not much solutions as PHP doesn't have explicit hash map data structures.

When using ArrayCollection you can use the ::toArray():

$arr = new Doctrine\Common\Collections\ArrayCollection();

$e1 = new stdclass();
$e2 = new stdclass();

$arr->add($e1);
$arr->add($e2);

$arr->remove(0);
json_encode($arr->toArray()) // => "{"1":{}}"
json_encode(array_values($arr->toArray())) // => "[{}]"

[Drachenkaetzchen]

@theofidry Yes, however, I believe this is really an issue DunglasApiBundle should solve (or a JsonEncoder which supports removing array keys). I even wonder if the hydra schema is violated with that behavior.

[theofidry]

Personally the solution I came up with was to do the array_values in the remove method, I think it's better in terms of performance.

I believe this is really an issue DunglasApiBundle should solve

I agree it would be great to address this issue. Maybe by adding a meta tag when we want to specify the collection manipulated is actually and hashmap or having an option to handle arrays as arrays only.

[dunglas]

This issue is related to Doctrine. It doesn't reindex arrays when a remove occurs for performance reasons but this lead to issues when using json_encode later (I first found this problem when using FOSRest + JMSSerializer).
The array_values option suggested by @theofidry is the better one IMO but it's not the responsibility of API Platform (it must be implemented in your domain code). However, it will be nice to have a doc explaining this edge case and the workaround.

[Drachenkaetzchen]

@dunglas sorry but I have to disagree. I can't see how removals from a collection and a resulting issue is a domain-specific edge case. One can't assume that only the last item of a collection gets removed.

Think of a blog with a 1:n comments relation: If the API user removes any comment but the last, the issue will occur.

[theofidry]

What do you think of having an annotation for hashmaps @dunglas? It's true that the root of the problem is Doctrine but they won't fix it and as @FELICITUS say, it's quite annoying as it occurs more often than not.

[dunglas]

@FELICITUS OK let me reword: it's a Doctrine problem. But I think they will not fix it. I think it's not the responsibility of API Platform to fix Doctrine bugs/edge cases (we are in the process of removing all hard dependencies to Doctrine and to support other persistence system such as POMM).

I'm not for adding such annotation, it will be a pain to maintain. What do you think about an entry in the doc?

[Drachenkaetzchen]

@dunglas Yes, with that I must agree. A doc entry is almost absolutely necessary. Especially if users will implement DunglasApiBundle with many entities (like I do).

[theofidry]

I'm not for adding such annotation, it will be a pain to maintain. What do you think about an entry in the doc?

That lots of people will still encounter the issue as we use almost always use arrays rather than hashmaps and I'm not sure POMM, Eloquent or Propel are careful of that as this issue is encountered only for JSON serialization.

[dunglas]

A fix can be an option in the Doctrine mapping (reorder: true or something similar). I doubt they will allow that but you can still open an issue on their bug tracker, maybe that it's a common problem.

My POV is that fixing an issue occurring in other contexts (FOSRest for instance) in API Platform is not right.
The real problem, as you pointed out, is the lack of real list structure in PHP and AFAIK this is not going to be fixed soon...

[theofidry]

Then leave a VERY BIG mention in the doc ;)

[Drachenkaetzchen]

My feeling is that this probably could be fixed in the Serializer somehow, as json_encode is most likely the culprit here due to non-consistent behavior.