IndexError: invalid index to scalar variable

hello guys !!
i try to run lda2vec algorithm with m own data from : lda2vec . i have one fichier csv with 3 data (idTweet,textTweet,dateTweet) and when i try to run the module , i’m getting this error :

n_words 21
[ 0 1 2 … 4324 4326 4325]
n_stories 4327
Traceback (most recent call last):
File “”, line 74, in
flattened, features_flat = corpus.compact_to_flat(pruned,*feature_arrs)
File “/usr/local/lib/python3.6/dist-packages/lda2vec-0.1-py3.6.egg/lda2vec/”, line 422, in compact_to_flat
IndexError: invalid index to scalar variable.
how to fix this error.
voici mon code :

from lda2vec import preprocess, Corpus

import numpy as np

import pandas as pd

import logging

import pickle

import os.path

import spacy

from spacy.attrs import LOWER


max_length = 250 # Limit of 250 words per comment

min_author_comments = 50 # Exclude authors with fewer comments

nrows = None # Number of rows of file to read; None reads in full file

nlp = spacy.load(“en_core_web_sm”)

#fn = “/content/drive/MyDrive/lda2vec-master/examples/hacker_news/data/mars_avril.csv”

features = []

Convert to unicode (spaCy only works with unicode)

features = pd.read_csv("/content/drive/MyDrive/lda2vec-master/examples/hacker_news/data/mars_avril.csv", encoding=‘utf8’, nrows=nrows)

Convert all integer arrays to int32

for col, dtype in zip(features.columns, features.dtypes):

if dtype is np.dtype('int64'):

    features[col] = features[col].astype('int32') 

Tokenize the texts

If this fails it’s likely spacy. Install a recent spacy version.

Only the most recent versions have tokenization of noun phrases

I’m using SHA dfd1a1d3a24b4ef5904975268c1bbb13ae1a32ff

Also try running python -m all --force

texts = features.pop(‘textTweet’).values

tokens, vocab = preprocess.tokenize(str(texts), max_length, skip=-2, attr=LOWER, merge=False, nlp=nlp)

del texts

Make a ranked list of rare vs frequent words

corpus = Corpus()



The tokenization uses spaCy indices, and so may have gaps

between indices for words that aren’t present in our dataset.

This builds a new compact index

compact = corpus.to_compact(tokens)

Remove extremely rare words

pruned = corpus.filter_count(compact, min_count=10)

Words tend to have power law frequency, so selectively

downsample the most prevalent words

clean = corpus.subsample_frequent(pruned)

print (“n_words”, np.unique(clean).max())

Extract numpy arrays over the fields we want covered by topics

Convert to categorical variables

#author_counts = features[‘dateTweet’].value_counts()

#to_remove = author_counts[author_counts < min_author_comments].index

#mask = features[‘dateTweet’].isin(to_remove).values

#author_name = features[‘dateTweet’].values.copy()

#author_name[mask] = ‘infrequent_author’

#features[‘dateTweet’] = author_name

#authors = pd.Categorical(features[‘dateTweet’])

#author_id =

#author_name = authors.categories

story_id = pd.Categorical(features[‘idTweet’]).codes

Chop timestamps into days


#story_time = pd.to_datetime(features[‘dateTweet’], unit=‘s’)

#days_since = (story_time - story_time.min()) / pd.Timedelta(‘1 day’)

#time_id = days_since.astype(‘int32’)

features[‘story_id_codes’] = story_id

#features[‘author_id_codes’] = story_id

#features[‘time_id_codes’] = time_id

#print (“n_authors”, author_id.max())

print (“n_stories”, story_id.max())

#print (“n_times”, time_id.max())

Extract outcome supervised features

#ranking = features[‘comment_ranking’].values

#score = features[‘story_comment_count’].values

Now flatten a 2D array of document per row and word position

per column to a 1D array of words. This will also remove skips

and OoV words

feature_arrs = (story_id)

flattened, features_flat = corpus.compact_to_flat(pruned,*feature_arrs)

Flattened feature arrays

story_id_f = features_flat

Save the data

pickle.dump(corpus, open(‘corpus’, ‘wb’), protocol=2)

pickle.dump(vocab, open(‘vocab’, ‘wb’), protocol=2)


data = dict(flattened=flattened, story_id=story_id_f,author_id=author_id_f)

np.savez(‘data’, **data)‘tokens’, ‘wb’), tokens)