Planet SciPy
pelas ruas de Belém
as vezes pelas ruas de Belém não sei se sou eu ou minha mãe quem está aliEnterprises Need to Think Differently about Data Science. Here’s How.
Companies that are data science literate make and communicate decisions on the basis of real data models, and not merely instinct or tradition. They welcome new data science technologies as opportunities for potential innovation, rather…
The post Enterprises Need to Think Differently about Data Science. Here’s How. appeared first on Anaconda.
Moving Conda Environments
Conda is known as a package manager for Python and R packages produced by Anaconda, Inc. and conda-forge, the open-source community for conda Python packages. In addition to managing packages, Conda is also an environment…
The post Moving Conda Environments appeared first on Anaconda.
Spyder 4.0 beta4: Kite integration is here
Kite is sponsoring the work discussed in this blog post, and in addition supports Spyder 4.0 development through a Quansight Labs Community Work Order.
As part of our next release, we are proud to announce an additional completion client for Spyder, Kite. Kite is a novel completion client that uses Machine Learning techniques to find and predict the best autocompletion for a given text. Additionally, it collects improved documentation for compiled packages, i.e., Matplotlib, NumPy, SciPy that cannot be obtained easily by using traditional code analysis packages such as Jedi.
Read more… (3 min remaining to read)
4 Machine Learning Use Cases in the Automotive Sector
From parts suppliers to vehicle manufacturers, service providers to rental car companies, the automotive and related mobility industries stand to gain significantly from implementing machine learning at scale. We see the big automakers investing in…
The post 4 Machine Learning Use Cases in the Automotive Sector appeared first on Anaconda.
An initial report on the Common Fund Data Ecosystem
Helping DCCs get FAIRer
Accessing Remote Data with a Generalized File System
Originally created for the needs of Dask, we have spun out a general file system implementation and specification, to provide all users with simple access to many local, cluster, and remote storage media. Dask and Intake…
The post Accessing Remote Data with a Generalized File System appeared first on Anaconda.
Getting Started with Machine Learning in the Enterprise
Machine learning (ML) is a subset of artificial intelligence (AI) in which data scientists use algorithms and statistical models to predict outcomes and/or perform specific tasks. ML models can automatically “learn from” data sets to…
The post Getting Started with Machine Learning in the Enterprise appeared first on Anaconda.
What’s in a Name? Clarifying the Anaconda Metapackage
The name “Anaconda” is overloaded in many ways. There’s our company, Anaconda, Inc., the Anaconda Distribution, the anaconda metapackage, Anaconda Enterprise, and several other, sometimes completely unrelated projects (like Red Hat’s Anaconda). Here we hope…
The post What’s in a Name? Clarifying the Anaconda Metapackage appeared first on Anaconda.
Python : Learn Object Oriented Programming in 3 Hours
In the above example, car is an object. Functions are called methods in OOP world. Characteristics are attributes (properties). Technically attributes are variables or values related to the state of the object whereas methods
Dask joins NumFOCUS Sponsored Projects
The post Dask joins NumFOCUS Sponsored Projects appeared first on NumFOCUS.
4 Ways Financial Firms Put Machine Learning to Work
Several industry giants in the finance sector are well on their way to implementing machine learning technology that improves operations and guides strategy in multiple departments. So far, machine learning algorithms are being used in…
The post 4 Ways Financial Firms Put Machine Learning to Work appeared first on Anaconda.
Precision Recall Curve Simplified
Meet our 2019 Google Summer of Code Students (Part 3)
The post Meet our 2019 Google Summer of Code Students (Part 3) appeared first on NumFOCUS.
Announcing the Anaconda 2019.07 Release
The midyear release of Anaconda Distribution is available for download. The Anaconda 2019.07 release focuses on Conda performance. We’ve heard your frustration about lengthy install times for packages when using Conda, and the Conda 4.7…
The post Announcing the Anaconda 2019.07 Release appeared first on Anaconda.
Meet our 2019 Google Summer of Code Students (Part 2)
The post Meet our 2019 Google Summer of Code Students (Part 2) appeared first on NumFOCUS.
Comparing two genome binnings quickly with sourmash
Comparing two sets of MAGs, for fun and profit!
Calculate KS Statistic with Python
In data science, it compares the cumulative distribution of events and non-events and KS is where there is a maximum difference between the two distributions. In simple words, it helps us to understand how well our predictive model is able to discriminate between events and
Python : Complete Guide to Date and Time Functions
- datetime
- date
- time
- timedelta
People who have no experience of working with real-world datasets might have not encountered date columns. They might be under impression that working with dates is rarely used and not so
Meet our 2019 Google Summer of Code Students (Part 1)
The post Meet our 2019 Google Summer of Code Students (Part 1) appeared first on NumFOCUS.
What are *args and **kwargs and How to use them
args is a short form of arguments. With the use of *args python takes any number of arguments in user-defined function and converts user inputs to a tuple named args. In other words, *args means zero or more arguments which are stored in a tuple named args. When you define function without *args, it has a fixed number of inputs which means it cannot accept more (or less) arguments than you defined in the function.
In the example code below, we are creating a very basic function which adds two numbers. At the same time, we created a
Quansight presence at SciPy'19
Yesterday the SciPy'19 conference ended. It was a lot of fun, and very productive. You can really feel that there's a lot of energy in the community, and that it's growing and maturing. This post is just a quick update to summarize Quansight's presence and contributions, as well as some of the more interesting things I noticed.
A few highlightsThe "Open Source Communities" track, which had a strong emphasis on topics like burnout, diversity and sustainability, as well as the keynotes by Stuart Geiger ("The Invisible Work of Maintaining and Sustaining Open-Source Software") and Carol Willing ("Jupyter: Always Open for Learning and Discovery") showed that many more people and projects are paying more attention to and evolving their thinking on the human and organizational aspects of open source.
I did not go to many technical talks, but did make sure to catch Matt Rocklin's talk "Refactoring the SciPy Ecosystem for Heterogeneous Computing". Matt clearly explained some key issues and opportunities around
Python : 10 Ways to Filter Pandas DataFrame
- Select all the active customers whose accounts were opened
Ibis: Python data analysis productivity framework
Ibis is a library pretty useful on data analysis tasks that provides a pandas-like API that allows operations like create filter, add columns, apply math operations etc in a lazy mode so all the operations are just registered in memory but not executed and when you want to get the result of the expression you created, Ibis compiles that and makes a request to the remote server (remote storage and execution systems like Hadoop components or SQL databases). Its goal is to simplify analytical workflows and make you more productive.
Ibis was created by Wes McKinney and is mainly maintained by Phillip Cloud and Krisztián Szűcs. Also, recently, I was invited to become a maintainer of the Ibis repository!
Maybe you are thinking: "why should I use Ibis?". Well, if you have any of the following issues, probably you should consider using Ibis in your analytical workflow!
- if you need to get data from a SQL database but you don't
Python Dictionary Comprehension with Examples
- Keys : Think about columns in tables. It must be unique (like column names cannot be duplicate)
- Values : It is similar to rows in tables. It can be duplicate.
To extract keys, values and structure of dictionary, you can submit the following commands.
d = {'a': [1,2], 'b': [3,4], 'c': [5,6]}
d.keys() # 'a', 'b', 'c'
d.values() # [1, 2], [3, 4], [5,
HTML outputs in Jupyter
SummaryUser interaction in data science projects can be improved by adding a small amount of visual deisgn.
To motivate effort around visual design we show several simple-yet-useful examples. The code behind these examples is small and accessible to most Python developers, even if they don’t have much HTML experience.
This post in particular focuses on Jupyter’s ability to add HTML output to any object. This can either be full-fledged interactive widgets, or just rich static outputs like tables or diagrams. We hope that by showing examples here we will inspire some throughts in other projects.
This post was supported by replies to this tweet. The rest of this post is just examples.
IrisI originally decided to write this post after reading another blogpost from the UK Met office, where they included the HTML output of their library Iris in a a blogpost
(work by Peter Killick, post by Theo McCaie)
The fact that the output provided by an interactive session is the same output that you would provide in a published result helps everyone. The interactive
(continued...)Why We Removed the “Free” Channel in Conda 4.7
One of the changes we made in Conda 4.7 was the removal of a software collection called “free” from the default channel configuration. The “free” channel is our collection of packages prior to the switch…
The post Why We Removed the “Free” Channel in Conda 4.7 appeared first on Anaconda.
Python list comprehension with Examples
f(x) = x2, f(x) will always return the same result with the same x value. The function has no "side-effect" which means an operation has no effect on a variable/object that is outside the intended usage. "Side-effect" refers to leaks in your code which can modify a mutable data structure or variable. Functional programming is also good for parallel computing as there is no
uarray update: API changes, overhead and comparison to __array_function__
uarray is a generic override framework for objects and methods in Python. Since my last uarray blogpost, there have been plenty of developments, changes to the API and improvements to the overhead of the protocol. Let’s begin with a walk-through of the current feature set and API, and then move on to current developments and how it compares to __array_function__. For further details on the API and latest developments, please see the API page for uarray. The examples there are doctested, so they will always be current.
NumPy is a simple, rectangular, dense, and in-memory data store. This is great for some applications but isn't complete on its own. It doesn't encompass every single use-case. The following are examples of array objects available today that have different features and cater to a different kind of audience.
- Dask is one of the most popular ones. It allows distributed and chunked computation.
- CuPy is another popular one, and
Don't cite the No Free Lunch Theorem
Tldr; You probably shouldn’t be citing the "No Free Lunch" Theorem by Wolpert. If you’ve cited it somewhere, you might have used it to support the wrong conclusion. What it actually (vaguely) says is “You can’t learn from data without making assumptions”.The paper on the “No Free Lunch Theorem”, actually called "The Lack of A Priori Distinctions Between Learning Algorithms" is one of these papers that are often cited and rarely read, and I hear many people in the ML community refer to it when supporting the claim that “one model can’t be the best at everything” or “one model won’t always be better than another model”. The point of this post is to convince you that this is not what the paper or theorem says (at least not the one usually cited by Wolpert), and you should not cite this theorem in this context; and also that common versions cited of the "No Free Lunch" Theorem (continued...)
15 ways to read CSV file with pandas
pip install pandas. Next step is to load the package by running the following command. pd is an alias of pandas package. We will use it instead of full name "pandas". import pandas as pd
Moderna joins NumFOCUS Corporate Sponsors
The post Moderna joins NumFOCUS Corporate Sponsors appeared first on NumFOCUS.
How We Made Conda Faster in 4.7
We’ve witnessed a lot of community grumbling about Conda’s speed, and we’ve experienced it ourselves. Thanks to a contract from NASA via the SBIR program, we’ve been able to dedicate a lot of time recently…
The post How We Made Conda Faster in 4.7 appeared first on Anaconda.
Matplotlib Tutorial : Learn with Examples in 3 hours
MATLAB programming language and provides a similar MATLAB like interface for graphics. The beauty of this library is that it integrates well with pandas package which is used for data manipulation. With the combination of these two libraries, you can easily perform data wrangling along with visualization and get valuable insights out of data. Like ggplot2 library in R, matplotlib library is the grammar of graphics in Python and most used library for charts in Python. 
Write Short Blogposts
I encourage my colleagues to write blogposts more frequently. This is for a few reasons:
-
It informs your broader community what you’re up to, and allows that community to communicate back to you quickly.
You communicating to the community fosters a sense of collaboration, openness, and trust. You gain collaborators, build momentum behind your work, and curate a body of knowledge that early adopters can consume to become experts quickly.
Getting feedback from your community helps you to course-correct early in your work, and stops you from wasting time in inefficient courses of action.
You can only work for a long time without communicating if you are either entirely confident in what you’re doing, or reckless, or both.
-
It increases your visibility, and so is good for your career.
I have a great job. I find my work to be both
How to encourage participation in teleconferences
Participation is good!
How to drop one or multiple columns in Pandas Dataframe
- Drop or Keep rows and columns
- Aggregate data by one or more columns
- Sort or reorder data
- Merge or append multiple dataframes
- String Functions to handle text data
- DateTime Functions to handle date or time format columns
import command. import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
A B C D
0 -1.236438 -1.656038
Google Renews Corporate Sponsorship of NumFOCUS
The post Google Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
NumFOCUS Hires First Ever Development Director
The post NumFOCUS Hires First Ever Development Director appeared first on NumFOCUS.
Labs update and May highlights
Time flies when you're having fun. Here is an update of some of the highlights of my second month at Quansight Labs.
The making of a black hole image & GitHub SponsorsBoth Travis and myself were invited by GitHub to attend GitHub Satellite in Berlin. The main reason was that Nat Friedman (GitHub CEO) decided to spend the first 20 minutes of his keynote to highlight the Event Horizon Telescope's black hole image and the open source software that made that imaging possible. This included the scientific Python very prominently - NumPy, Matplotlib, Python, Cython, SciPy, AstroPy and other projects were highlighted. At the same time, Nat introduced new GitHub features like "used by", a triaging role and new dependency graph features and illustrated how those worked for NumPy. These features will be very welcome news to maintainers of almost any project.
The single most visible feature introduced was GitHub Sponsors:
I really enjoyed meeting Devon Zuegel, Product Manager of the Open Source Economy Team at GitHub, in person after previously having had
SciPy and MathJax join NumFOCUS Sponsored Projects
The post SciPy and MathJax join NumFOCUS Sponsored Projects appeared first on NumFOCUS.
String Functions in Python with Examples
The cost of an open source contribution
Open source is massively successful. Some say it’s eating the world, although to my ears that phrasing doesn’t sound entirely like a good thing. Open source maintainers are always in need of help, and over the past years I’ve seen a lot of focus on ways open source projects can grow their communities and gain new contributors. Guidance on how to go about finding new contributors is easily found. E.Anaconda Recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms
The Anaconda team is excited to announce that we have been recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms. According to Gartner, “The Gartner Peer Insights…
The post Anaconda Recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms appeared first on Anaconda.
TDK-Micronas partners with Quansight to sponsor Spyder
TDK-Micronas is sponsoring Spyder development efforts through Quansight Labs. This will enable the development of some features that have been requested by our users, as well as new features that will help TDK develop custom Spyder plugins in order to complement their Automatic Test Equipment (ATE’s) in the development of their Application Specific Integrated Circuits (ASIC’s).
At this point it may be useful to clarify the relationship the role of Quansight Labs in Spyder's development and the relationship with TDK. To quote Ralf Gommers (director of Quansight Labs):
"We're an R&D lab for open source development of core technologies around data science and scientific computing in Python. And focused on growing communities around those technologies. That's how I see it for Spyder as well: Quansight Labs enables developers to be employed to work on Spyder, and helps with connecting them to developers of other projects in similar situations. Labs should be an enabler to let the Spyder project, its community and individual developers grow. And Labs provides mechanisms to attract and coordinate funding. Of course
metadsl: A Framework for Domain Specific Languages in Python
metadsl: A Framework for Domain Specific Languages in Python¶Hello, my name is Saul Shanabrook and for the past year or so I have been at Quansight exploring the array computing ecosystem. This started with working on the xnd project, a set of low level primitives to help build cross platform NumPy-like APIs, and then started exploring Lenore Mullin's work on a mathematics of arrays. After spending quite a bit of time working on an integrated solution built on these concepts, I decided to step back to try to generalize and simplify the core concepts. The trickiest part was not actually compiling mathematical descriptions of array operations in Python, but figuring out how to make it useful to existing users. To do this, we need to meet users where they are at, which is with the APIs they are already familiar with, like numpy. The goal of metadsl is to make it easier to tackle parts
Community-driven open source and funded development
Quansight Labs is an experiment for us in a way. One of our main aims is to channel more resources into community-driven PyData projects, to keep them healthy and accelerate their development. And do so in a way that projects themselves stay in charge.
This post explains one method we're starting to use for this. I'm writing it to be transparent with projects, the wider community and potential funders about what we're starting to do. As well as to explicitly solicit feedback on this method.
Community work ordersIf you talk to someone about supporting an open source project, in particular a well-known one that they rely on (e.g. NumPy, Jupyter, Pandas), they're often willing to listen and help. What you quickly learn though is that they want to know in some detail what will be done with the funds provided. This is true not only for companies, but also for individuals. In addition, companies will likely want a written agreement and some form of reporting about the progress of the work. To meet this
Why citations are not enough for open source software
A few weeks ago I wrote about why you should cite open source tools. Although I think citations important, though, there are major problems in relying on them alone to support open source work.
The biggest problem is that papers describing a software library can only give credit to the contributors at the time that the paper was written. The preferred citation for the SciPy library is “Eric Jones, Travis Oliphant, Pearu Peterson, et al”, 2001. The “et al” is not an abbreviation here, but a fixed shorthand for all other contributors. Needless to say many, many people have contributed to the SciPy library since 2001 (GitHub counts 716 contributors as of this writing), and they are unable to get credit within the academic system for those contributions. (As an aside, Google counts about 1,200 citations to SciPy, which is a breathtaking undercounting of its value and influence, and reinforces my earlier point: cite open source software! Definitely don't use this post as an excuse not to cite it!!!)
Not surprisingly, we have had
Measuring API usage for popular numerical and scientific libraries
Developers of open source software often have a difficult time understanding how others utilize their libraries. Having better data of when and how functions are being used has many benefits. Some of these are:
- better API design
- determining whether or not a feature can be deprecated or removed.
- more instructive tutorials
- understanding the adoption of new features
We wrote a general tool
python-api-inspect
to analyze any function/attribute call within a given set of
namespaces in a repository. This work was heavily inspired by a blog
post on inspecting method usage with
Google BigQuery
for pandas,
NumPy, and
SciPy. The previously mentioned work used
regular expressions to search for method usage. The primary issue with
this approach is that it cannot handle import numpy.random as rand;
rand.random(...) unless additional regular expressions are
constructed for each case and will result in false
positives. Additionally,
BigQuery is not a free resource.
Thus, this approach is not general enough and does not scale well with
the number of libraries that we would like to inspect function and
attribute usage.
A more robust
Intake: Discovering and Exploring Data in a Graphical Interface
Motivation Do you have data that you’d like people to be able to explore on their own? Are you always passing around snippets of code to load specific data files? These are problems that people…
The post Intake: Discovering and Exploring Data in a Graphical Interface appeared first on Anaconda.
Spyder 4.0 takes a big step closer with the release of Beta 2!
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been working hard on our next release. Over the past year, we've
Spyder 4.0 takes a big step closer with the release of Beta 2!
This blogpost was originally published on the Quansight Labs website
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been
(continued...)The Role of a Maintainer
What are the expectations and best practices for maintainers of open source software libraries? How can we do this better?
This post frames the discussion and then follows with best practices based on my personal experience and opinions. I make no claim that these are correct.
Let us Assume External ResponsibilityFirst, the most common answer to this question is the following:
- Q: What are expectations on OSS maintainers?
- A: Nothing at all. They’re volunteers.
However, let’s assume for a moment that these maintainers are paid to maintain the project some modest amount, like 10 hours a week.
How can they best spend this time?
What is a Maintainer?Next, let’s disambiguate the role of developer, reviewer, and maintainer
-
Developers fix bugs and create features. They write code and docs and generally are agents of change in a software project. There are often many more developers than reviewers or maintainers.
-
Reviewers are known
Two Sigma Renews Corporate Sponsorship of NumFOCUS
The post Two Sigma Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
Using GitHub for janky project reporting - some code
We scripted GitHub for lightweight project reporting
My first DNF (Ft Bragg 600k)
It's been six years since my first ride with The San Francisco Randonneurs and four years since my first 200k. I've ridden 18 rides that are at least that distance since then (3x 300k, 2x 400k, 1x 600x), completing my first Super Randonneur Series (2-, 3-, 4-, and 600k in one year) last year after not riding much the year before that. And this weekend I had my first DNF result on the Fort Bragg 600k. I Did Not Finish.
The best response to my choice of abandoning the ride to enjoy the campground came from Peter Curley, who said "That was a very mature decision." A clear departure from typical randonneuring stubbornness and refusal to give up, I celebrated my decision to quit as a victory when I arrived at the campground and made my announcement to the volunteers. I think I was so energetic about it that they did not believe me. I was being kind to myself, to my body, and at peace with the decision by
(continued...)Should I Resign from My Full Professor Job to Work Fulltime on Cocalc?
PyCon2019 poem
I'm back in Cleveland for another Pycon. Yesterday was my first full day here. Along with Matt Seale, I was a helper at Matthias Bussonnier tutorial ("IPython and Jupyter in Depth: High productivity, interactive Python). The sticky system is efficient at signaling when someone in a classroom needs help, and a lot of folks don't know that this practice was popularized by Software Carpentry workshops and continues to be used at The Carpentries.
I stepped out for a coffee refill and bumped into a large contingent of Bloomberg folks I'd never met (Princeton office). I guess we have something like 90 people at the conference this year, and I made the usual and true remark about how I go to conferences to meet the other people who work at our company. Then after his tutorial concluded, Matthias and I bumped into Tracy Teal, exchanged some stickers, and chatted about The Carpentries, Jupyter, organizing conferences, governance and sponsorship models, and a bunch of other stuff.
Matthias was a
(continued...)Why you should cite open source tools
Every now and then, a moment or a sentence in a conversation sticks out at you, and lodges itself in the back of your brain for months or even years. In this case, the sentence is a tweet, and I fear that the only way to dislodge it is to talk about it publicly.
Last year, I complained on Twitter that a very prominent paper that was getting lots of attention used scikit-image, but failed to cite our paper. (Or the papers corresponding to many other open source packages.) I continued that scientists developing open source software depend on these citations to continue their work. (More on this in another post...) One response was that surely the developers of the open source scientific Python stack were not scientists per se, and that citations were not a priority for them.
I still sigh internally when I think of it.
That tweet manifests a pervasive perception that open source scientific software is written by God-like figures. These massively experienced software developers have easy access to funds
IBM Renews Corporate Sponsorship of NumFOCUS
The post IBM Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
DISC Committee Nominations are open for 2019-2021 positions
The post DISC Committee Nominations are open for 2019-2021 positions appeared first on NumFOCUS.
Python Lambda Function with Examples
anonymous function as the function can be defined without its name. They are a part of functional programming style which focus on readability of code and avoids changing mutable data. 
lambda arguments: expressionLambda function can have more than one argument but expression cannot be more than 1. The expression is evaluated and returned. Example
addition = lambda x,y: x + yIn the above python code,
addition(2,3) returns 5
x,y are the arguments and x + y is the expression that gets evaluated and returned. NumFOCUS Projects Accepted to Inaugural Google Season of Docs
The post NumFOCUS Projects Accepted to Inaugural Google Season of Docs appeared first on NumFOCUS.
Loops in Python explained with examples
Real World Examples of Loop
- Software of the ATM machine is in a loop to process transaction after transaction until you acknowledge that you have no more to do.
- Software program in a mobile device allows user to unlock the mobile with 5 password attempts. After that it resets mobile device.
- You put your favorite song on a repeat mode. It is also a loop.
- You want to run a particular analysis on each column of your data
Some questions and thoughts on journal peer review.
What's up with current peer review practice?
Create Dummy Data in Python
DataFrame functionimport pandas as pd
data = pd.DataFrame({"A" : ["John","Deep","Julia","Kate","Sandy"],
"MonthSales" : [25,30,35,40,45]})
A MonthSales
0 John 25
1 Deep 30
2
Things to think about when developing shotgun metagenome classifiers
Thoughts on goals and tradeoffs in classifying shotgun metagenome data.
Our submission for the NHGRI Human Genome Reference Center call
We wrote a big grant proposal!
PIP connection Error : SSL CERTIFICATE VERIFY FAILED
Error looks like this :
Could not fetch URL https://pypi.python.org/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)
 |
| PIP SSL Certification Issue |
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name> -vvv
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org pandas -vvvInstall Python Package
Ways to Install Python Package
If Anaconda is installed on your system (laptop), click on Anaconda Prompt as shown in the image below. |
| Anaconda Prompt |
To install a python package or module, enter the code below in Anaconda Prompt -
pip install package-name
 |
| Install |
News from the NIH Data Commons Pilot Phase Consortium
The NIH Data Commons Pilot Phase Consortium is dead! (Long live the NIH Data Commons!)
Critically assessing open science - the CAOS meeting.
A summary of the CAOS open science meeting
Computational Psychiatry: Combining multiple levels of analysis to understand brain disorders - PhD thesis
I noticed that as my personal website at my former university went down that my PhD thesis could not be found anywhere, so I'm posting it here.
During my PhD I explored how machine learning and computational modeling of the brain can be used to improve our understanding, and diagnostics …
NIXML: nix + YAML for easy reproducible environments
The rise and fall of bioconda A year ago, I remember a conversation which went basically like this: Them: So, to distribute my package, what do you think I should use? Me: You should use bioconda. Them: OK, that’s interesting, but what about …? Me: No, you should use bioconda. Them: I will definitely look … Continue reading NIXML: nix + YAML for easy reproducible environments →Sustaining open source: thinking about communities of effort
Thinking about how to sustain open source.
My recent reading re sustaining open communities
What has Titus been reading lately?
Reduzindo a pilha
Sou fã de quadrinhos desde criança. As primeiras revistas que ganhei foram na primeira metade dos anos 90, alguns Mickeys, Mônicas, Trapalhões e X-Men. Em 98 comecei a comprar X-Men, Fabulosos X-Men e Wolverine, até os primeiros números da famigerada X-Men Premium. Sem dinheiro, enveredei pelos mangás e histórias fechadas. Quando a Panini começa a... [Read More]Threat models for open online scientific engagement?
What threats are there for scientists in engaging in open online discussions?
Packaging PyQt5 apps with fbs — Distribute cross-platform GUI applications with the fman Build System
fbs is a cross-platform PyQt5 packaging system which supports building desktop applications for Windows, Mac and Linux (Ubuntu, Fedora and Arch). Built on top of PyInstaller it wraps some of the rough edges and defines a standard project structure which allows the build process to be entirely automated. The included …
Announcement: Audio TK 3.1.0
ATK is updated to 3.1.0 with heavy code refactoring. Old C++ standards are now dropped and it requires now a full C++17 compliant compiler. The main difference for filter support is that explicit SIMD filters using libsimdpp have been dropped while tr2::simd becomes standard and supported by gcc, clang and Visual Studio. Download link: ATK […]Telemulta
A voz feminina robótica (chegamos no tempo onde questão de gênero e robôs podem se confundir) soou, estranha e familiar como sempre, assim que o carro finalizou a curva para a direita: “Você entrou na Avenida Universitária; o limite de velocidade é 60 quilômetros por hora”. Meu pai sorriu e começou a falar: – Desde... [Read More]My foreword to "Bayesian Analysis with Python, 2nd Edition" by Osvaldo Martin
When Osvaldo asked me to write the foreword to his new book I felt honored, excited, and a bit scared, so naturally I accepted. What follows is my best attempt to convey what makes probabilistic programming so exciting to me. Osvaldo did a great job with the book, it is …
Call for Answers: Survey About Task Assignment
Professor Igor Steinmacher, from Northern Arizona University, is a proeminent researcher on several social dynamics in open source communities, like support of newcomers, gender bias, open sourcing proprietary software, and more. Some of his papers can de found in his website. Currently, Prof. Igor is inviting mentors from open source communities to answer a survey... [Read More]Revisiting authorship, and JOSS software publications
The question du jour: how should authorship on software papers be decided?
Using Bayesian Decision Making to Optimize Supply Chains
(c) 2019 Thomas Wiecki & Ravin Kumar
As advocates of Bayesian statistics in data science we often have to convince business-minded colleagues or customers of the added value of such an approach. While there are many good reasons for applying Bayesian modeling to solve business problems (Sean J Taylor recently had …
Mestrado em Ciência da Computação na UFPA 2019: Inteligência Computacional para Smart Grids; Metaheurísticas
Está aberto o processo seletivo para o mestrado em ciência da computação do PPGCC-UFPA. Nesse certame, estou disponibilizando 2 vagas para alunos que desenvolverão seus trabalhos junto aos demais pesquisadores no LAAI. As vagas são voltadas para os temas de inteligência computacional aplicada a Smart Grids e estudos sobre métodos metaheurísticos de otimização. Gostaria de... [Read More]LaKademy 2018
Em outubro de 2018, Florianópolis foi sede da sexta edição do LaKademy, o sprint latinoamericano do KDE. Esse momento é uma oportunidade para termos em um mesmo lugar vários desenvolvedores do KDE – tanto veteranos quanto novatos – de diferentes projetos para melhorarem os respectivos softwares em que trabalham e também planejar as ações de... [Read More]LaKademy 2018
Past October 2018, Florianópolis hosted the 6th edition of LaKademy, the Latin-American KDE sprint. That moment is an opportunity to put together several KDE developers – both veterans and newcomers – from different projects in order to work for improve their respective software and plan the promotional actions of the community in the subcontinent. In... [Read More]GPU Dask Arrays, first steps
The following code creates and manipulates 2 TB of randomly generated data.
import dask.array as da
rs = da.random.RandomState()
x = rs.normal(10, 1, size=(500000, 500000), chunks=(10000, 10000))
(x + 1)[::2, ::2].sum().compute(scheduler='threads')
On a single CPU, this computation takes two hours.
On an eight-GPU single-node system this computation takes nineteen seconds.
Combine Dask Array with CuPyActually this computation isn’t that impressive. It’s a simple workload, for which most of the time is spent creating and destroying random data. The computation and communication patterns are simple, reflecting the simplicity commonly found in data processing workloads.
What is impressive is that we were able to create a distributed parallel GPU array quickly by composing these three existing libraries:
-
CuPy provides a partial implementation of Numpy on the GPU.
-
Dask Array provides chunked algorithms on top of Numpy-like libraries like Numpy and CuPy.
This enables us to operate on more data than we could fit in memory by operating on that data in
Manage project dependencies with conda environments
TL;DR: Create a conda environment for each project, capture exact versions when possible, automate activation and updating with a bash function.
I often work on several different projects involving software: Python libraries, papers, presentations, posters, this website, etc. Each project has different dependencies and there is a non-zero chance that these dependencies might be in conflict with each other. For example, I need Python 2.7 to work on a tesseroid modeling paper with a student, while my current work on
(continued...)First Impressions of GPUs and PyData
I recently moved from Anaconda to NVIDIA within the RAPIDS team, which is building a PyData-friendly GPU-enabled data science stack. For my first week I explored some of the current challenges of working with GPUs in the PyData ecosystem. This post shares my first impressions and also outlines plans for near-term work.
First, lets start with the value proposition of GPUs, significant speed increases over traditional CPUs.
GPU PerformanceLike many PyData developers, I’m loosely aware that GPUs are sometimes fast, but don’t deal with them often enough to have strong feeling about them.
To get a more visceral feel for the performance differences, I logged into a GPU machine, opened up CuPy (a Numpy-like GPU library developed mostly by Chainer in Japan) and cuDF (a Pandas-like library in development at NVIDIA) and did a couple of small speed comparisons:
Compare Numpy and Cupy>>> import numpy, cupy
>>> x = numpy.random.random((10000, 10000))
>>> y = cupy.random.random((10000, 10000))
>>> %timeit bool((numpy.sin(x) ** 2 + numpy.cos(x) ** 2 == 1).all())
446 ms ± 53.1 ms perA quick read of _The genomic and proteomic landscape of the rumen microbiome_
Using short and long reads to assemble genomes from metagenomes!
Support Python 2 with Cython
SummaryMany popular Python packages are dropping support for Python 2 next month. This will be painful for several large institutions. Cython can provide a temporary fix by letting us compile a Python 3 codebase into something usable by Python 2 in many cases.
It’s not clear if we should do this, but it’s an interesting and little known feature of Cython.
Background: Dropping Python 2 Might be Harder than we ExpectMany major numeric Python packages are dropping support for Python 2 at the end of this year. This includes packages like Numpy, Pandas, and Scikit-Learn. Jupyter already dropped Python 2 earlier this year.
For most developers in the ecosystem this isn’t a problem. Most of our packages are Python-3 compatible and we’ve learned how to switch libraries. However, for larger companies or government organizations it’s often far harder to switch. The PyCon 2017 keynote by Lisa Guo and Hui Ding from Instagram gives a good look into why this can be challenging for large production codebases and also gives a good
(continued...)Anatomy of an OSS Institutional Visit
I recently visited the UK Meteorology Office, a moderately large organization that serves the weather and climate forecasting needs of the UK (and several other nations). I was there with other open source colleagues including Joe Hamman and Ryan May from open source projects like Dask, Xarray, JupyterHub, MetPy, Cartopy, and the broader Pangeo community.
This visit was like many other visits I’ve had over the years that are centered around showing open source tooling to large institutions, so I thought I’d write about it in hopes that it helps other people in this situation in the future.
My goals for these visits are the following:
- Teach the institution about software projects and approaches that may help them to have a more positive impact on the world
- Engage them in those software projects and hopefully spread around the maintenance and feature development burden a bit
We were invited by early adopters within the institution, both within the UK Met Office’s Informatics Lab
(continued...)The story behind the spacegraphcats project and paper
The TRUE story!
Notes on the Frank-Wolfe Algorithm, Part II: A Primal-dual Analysis
This blog post extends the convergence theory from the first part of my notes on the Frank-Wolfe (FW) algorithm with convergence guarantees on the primal-dual gap which generalize and strengthen the convergence guarantees obtained in the first part.
MathJax.Hub.Config({ extensions: ["tex2jax.js"], jax: ["input/TeX", "output/HTML-CSS"], tex2jax …Creating a welcoming teaching/learning environment in workshops
It takes constant work to make a welcoming teaching/learning environment!
Repeatability in Practice (2018 version)
How we do repeatability in the DIB Lab