logo
down
shadow

Masked Numpy arrays much slower than normal numpy arrays


Masked Numpy arrays much slower than normal numpy arrays

By : Lukas Brunner
Date : November 21 2020, 07:35 AM
I wish this help you EDIT: The original answer doesn't work for numpy versions <= 1.8, in which np.nansum([NaN, NaN]) == 0.0 (note the FutureWarning). For earlier versions, you'll have to check that case manually:
code :
tmp = 3.25 * np.nansum(rs) +  .75 * np.nansum(rs * rs)
if not np.isnan(tmp):
  runningSum += tmp
runningSum += 3.25 * np.sum(rs) +  .75 * np.dot(rs, rs)
runningSum += 3.25 * np.nansum(rs) +  .75 * np.nansum(rs * rs)
In [63]: genotypes = np.random.binomial(3, .333, size=(5000, 200)).astype(float)

In [64]: pValues = np.random.uniform(0,1,5000)

In [65]: %timeit testMask(pValues, genotypes)
1 loops, best of 3: 11.3 s per loop

In [66]: %timeit testNotMask(pValues, genotypes)
1 loops, best of 3: 3.53 s per loop

In [67]: %timeit fastNotMask(pValues, genotypes)
1 loops, best of 3: 3.96 s per loop

In [68]: randjs = np.random.randint(0,200, 10)

In [69]: randis = np.random.randint(0,5000,10)

In [70]: genotypes[randis,randjs] = None

In [71]: %timeit testMask(pValues, genotypes)
1 loops, best of 3: 33 s per loop

In [72]: %timeit testNotMask(pValues, genotypes)
1 loops, best of 3: 3.6 s per loop

In [73]: %timeit fastNotMask(pValues, genotypes)
1 loops, best of 3: 3.98 s per loop

In [74]: testMask(pValues, genotypes)
Out[74]: 0.47606794747438386

In [75]: testNotMask(pValues, genotypes)
Out[75]: nan

In [76]: fastNotMask(pValues, genotypes)
Out[76]: 0.47613597091679449


Share : facebook icon twitter icon
numpy.interp & masked arrays

numpy.interp & masked arrays


By : user3394115
Date : March 29 2020, 07:55 AM
it helps some times you can get the mask of input_array first, and apply it to the result array, and use scipy.stats.norm to calculate cdf of normal distribution, or you can use scipy.special.erf() to calculate cdf by using the cdf formula of normal distribution:
code :
import scipy.stats as stats    
def gaussian_stretch2(input_array, array_mean, array_standard_deviation, n):
    mask = input_array.mask
    n = stats.norm(array_mean, array_standard_deviation*n)
    return numpy.ma.array(n.cdf(input_array), mask=mask)
Using masked numpy arrays with rpy2

Using masked numpy arrays with rpy2


By : Cream Pineapple
Date : March 29 2020, 07:55 AM
may help you . The concept of "masked values" (that is of an array of value coupled to a list of indices to be masked) does not directly exist in R.
In R values are either set to be "missing" (NA), or a subset of the original data structure is taken (so a new object containing only this subset is created).
Does numpy.ma allow masking of sub-masked arrays in a masked array?

Does numpy.ma allow masking of sub-masked arrays in a masked array?


By : G Dheeraj
Date : March 29 2020, 07:55 AM
Hope this helps I did fix my problem by actually thinking about what I was passing. The shape of the sub-array was something like (A,B,C,D...). What you pass a mask, it should be of the same structure, so I was trying to assign True to a tuple. When I tried using (True,True,True...) it worked!
Add together two numpy masked arrays

Add together two numpy masked arrays


By : Arpad Sabo
Date : March 29 2020, 07:55 AM
like below fixes the issue Try this (choosing the logical operator that you want to use for your masks from http://docs.python.org/3/library/operator.html)
code :
>>> from operator import and_
>>> np.ma.array(arr1.data+arr2.data,mask=map(and_,arr1.mask,arr2.mask))
masked_array(data = [2 4 --],
             mask = [False False  True],
       fill_value = 999999)
>>> np.ma.array(arr1.data+arr2.data,mask=list(map(and_,arr1.mask,arr2.mask)))
Why are Numpy masked arrays useful?

Why are Numpy masked arrays useful?


By : STLShawn
Date : March 29 2020, 07:55 AM
I hope this helps you . The official answer is reported here:
code :
import numpy as np
g = np.random.random((5000,5000))
indx = np.random.randint(0,4999,(500,2))
g_nan = g.copy()
g_nan[indx] = np.nan
mask =  np.full((5000,5000),False,dtype=bool)
mask[indx] = True
g_mask = np.ma.array(g,mask=mask)

%timeit (g_mask + g_mask)**2
1.27 s ± 35.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
(g_nan + g_nan)**2
%timeit (g_nan + g_nan)**2
76.5 ms ± 715 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Related Posts Related Posts :
  • Filtering from data
  • Where is the problem about selenium with python?
  • ansible custom filter fails when importing python library
  • How to assign the label of one column to the new one based on group maximum in pandas
  • What is the best approach for isolating a single area of similar colour?
  • Creating multiple clients for topics
  • Why is my 'for loop', despite iterating over all keys, only acting on the last one?
  • Can someone tell me what's wrong, when I run it the browsers says "This site can’t be reached"
  • Error in setting up mitmproxy on alpine 3.9
  • From traditional loop to list comprehension
  • Django celery unregistered task | relative imports
  • How to add elements in a multi dimensional array
  • Async await with sqs receive messages not working properly
  • What is definition of 'NAME' in Python grammar
  • Easy method to move rows from df to another with coditions?
  • Changing the size of only a single plot in matplotlib, without altering figure parameters
  • Fastest way to use Vision API on 10,000+ images with python
  • How to install nvidia apex on Google Colab
  • Random numbers Continuous in python
  • Fetching data after a certain time interval(10 sec) from a continuously increasing database like mysql using flask
  • Using VLOOKUP with merge in Python
  • Calculate geographical distance between 5 cities with all the possible combinations of each city
  • How to filter a pandas dataframe using multiple partial strings?
  • Pygame- make bullet shoot toward cursor direction
  • Create SEQUENCE based dictionary from list
  • How to fix broken link from Django MEDIA_ROOT?
  • How can I display the current time left in a timer in a label?
  • Compute number of occurance of each value and Sum another column in Pandas
  • How to separate the prefix in words that are 'di'?
  • Handling network errors from an external API across an application
  • Want a pandas Series of Trips Completed to count(Request) ratio for each hour as index for the given dataframe
  • Access dict keys and list elements by same index to loop over and assign values
  • Find rows from the same dataframe based on condition
  • Read only specific part first two lines from text file in python
  • Python How to convert string to dataframe?
  • How to fix this my error code program? I use Python 3.6
  • Is there a way of getting this string down to 3 words?
  • Large difference between overall F Score for a custom Spacy NER model and Individual Entity F Score
  • Drop rows where timestamps are older than subsequent row
  • Implement a bottle spin
  • Unable to convert widows epoch time to normal date time
  • Values from a XML file
  • PyAudio readframes not ending when wav file completes
  • Could not load the module
  • How to change datetime.datetime(2012, 1, 1, 0, 0) to 1/1/2012 in Python?
  • How to create ASN.1 Sequence without NamedType?
  • How to locate specific sequences of words in a sentence efficiently
  • How can I generate a multi-step process in Django without changing pages (w/out a new request)?
  • Why does this list comprehension only "sometimes" work?
  • send html report with row collapsed
  • How to define a type hint to a argument (the argument's value is a class, all expected value is a subclass of a certain
  • How do I send a styled pandas DataFrame by e-mail without losing the format?
  • How to view/average a groupby dataframe when the data is a string?
  • Django 2.2 staticfiles do not work in development
  • Flag to enable/disable numba JIT compilation?
  • Trying to split byte in a byte array into two nibbles
  • Error in Query - missing FROM-clause entry for table - SQL
  • Reading double c structures of Dll with Ctypes in Python
  • Autofill missing row in database based on missing time range
  • Get the max of a nested dictionary
  • shadow
    Privacy Policy - Terms - Contact Us © festivalmusicasacra.org