2D to 3D: Creating Coordinated Items with CSM

August 23, 2023

In the previous blog post, we introduced CSM (Common Sense Machines) and compared their various Image to 3D services.

Using this service, I generated items such as shoes and bags, and tried coordinating them ☺.

To begin with, the process was simple, but it was quite challenging!

Procedure

Generate images with Midjourney.
Generate models with CSM.
Arrange them with Blender.

(Note: For this case, The clothes were created using Marvelous Designer.)

Generate images with Midjourney

When generating images of shoes, I used the prompt "side view".

Additionally, I experimented with whether a CG-like image would be generated more successfully than a photorealistic one. For this, I tried prompts like "3dmodel" or "3dcg".

For shoes that were placed side by side, as they would always generate symmetrical and sticking models, it was better to use images of just one shoe.

a sneaker from side view, 3dmodel, in the style of lo-fi aesthetics,Y2K,minimalistic,grunge fashion --v 5

When generating images of bags, I used the prompt "front view".

a bag from front view, 3dmodel, in the style of lo-fi aesthetics,Y2K,minimalistic,grunge fashion,voluminous forms --v 5

※ Some images may not be able to generate, so it is recommended to have multiple images prepared.

Generate models with CSM

As a side note... a new generation method has been added since the previous update ☺

Previously, only uploads from Discord were possible, but now you can upload from your own page after logging in, and manage your models there, making it very convenient!

This time, both the generated models for shoes and bags had quite a few flaws, and it was quite a struggle.

Regarding shoes, although they were generated smoothly in the previous attempt, this time, after inputting several images, it seemed that only one usable model was created.

For bags, the handles often disappeared, and miraculously, only one model with handles was successfully generated.

Arrange with Blender

I will arrange the shoes and bags on an avatar.

Due to the convenience of the task this time, I performed vertex merging, but we only adjusted the scale and placement without touching the mesh in particular.

As the materials originally had strong contrast, I made some adjustments. The nodes were set up as follows.

Result

Here is the actual rendering of the generated items.

Originally, I wanted to create several coordinated outfits, but even though I generated about 10 images each for shoes and bags, only one model each seemed usable in reality.

If you plan to use them as coordinated items, it is desirable to have more variations... However, this time, there was only this one coordination.

Regarding the materials, even though I made adjustments to both models, they differ quite a bit from the impression of the input images.

Summary

I tried to create outfits using the models made by CSM.

I struggled a lot with generating models that had fewer flaws this time.

Regarding this point, it was an unexpected result, as in the previous comparison of shoes, CSM had a high accuracy, which was the motivation for trying it out.

However, after going through the rendering process this time, I also felt a slight expectation that it could be relatively easy to handle if the placement and simple material adjustments were all that was needed.

Future challenges may include the following points:

What types of input images have a higher probability of generating successful results?
What are the strengths and weaknesses of CSM?

Furthermore, there are significant color differences and difficulties in reproducing the texture between the input images and the generated models, so it would be good to explore methods for effectively adjusting those areas.