www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - OT: why do people use python when it is slow?

reply Laeeth Isharc <laeethnospam nospamlaeeth.com> writes:
https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
Andrei suggested posting more widely.
Oct 13 2015
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


That's flaimbait:

«Many really popular websites use Python. But why is that? 
Doesn't it affect the performance of the website?»

No. Really popular websites use pre-generated content / front end 
caches / CDNs or wait for network traffic from distributed 
databases.
Oct 13 2015
next sibling parent reply Laeeth Isharc <nospamlaeeth nospamlaeeth.com> writes:
On Wednesday, 14 October 2015 at 05:42:12 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 That's flaimbait:

 «Many really popular websites use Python. But why is that? 
 Doesn't it affect the performance of the website?»

 No. Really popular websites use pre-generated content / front 
 end caches / CDNs or wait for network traffic from distributed 
 databases.


For a long time, Ola, I am done discussing with you.  But I would 
ask you to take more responsibility for the effect of you words.  
The piece you quote is from the question, and not from what I 
wrote.  You refer to it as flame bait, and don't make this clear 
in what you write.
Oct 14 2015
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Wednesday, 14 October 2015 at 15:54:49 UTC, Laeeth Isharc 
wrote:

 For a long time, Ola, I am done discussing with you.  But I 
 would ask you to take more responsibility for the effect of you 
 words.  The piece you quote is from the question, and not from 
 what I wrote.  You refer to it as flame bait, and don't make 
 this clear in what you write.


The question as phrased is flame bait and trolling because the 
answer is obvious and generates lots of noise for no reason.

Latency is not affected by a script that does very little work 
compared to all the other causes for latency in a complex 
architecture. Including memcache access, database retrieval and 
compression.

What is affected by using Python over Go/D is the number of 
instances that run the service.

But for a well designed architecture up to 99% of the work is 
done by specialized infrastructure, implemented in 
Java/Erlang/C++, that often is too costly to develop for a single 
project. So you use ready-mades.

Low latency and scaling is a result of architecture, not brute 
force computation.
Oct 14 2015
prev sibling parent reply Mengu <mengukagan gmail.com> writes:
On Wednesday, 14 October 2015 at 05:42:12 UTC, Ola Fosheim 
Grøstad wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 That's flaimbait:

 «Many really popular websites use Python. But why is that? 
 Doesn't it affect the performance of the website?»

 No. Really popular websites use pre-generated content / front 
 end caches / CDNs or wait for network traffic from distributed 
 databases.


really popular portals, news sites? yes. really popular websites? 
nope. like booking.com, airbnb.com, reddit.com are popular 
websites that have many parts which have to be dynamic and 
responsive as hell and they cannot use caching, pre-generated 
content, etc.

using python affect the performance of your website. if you were 
to use ruby or php your web app would be slower than it's python 
version. and python version would be slower than go or d version.
Oct 14 2015
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:

 websites? nope. like booking.com, airbnb.com, reddit.com are 
 popular websites that have many parts which have to be dynamic 
 and responsive as hell and they cannot use caching, 
 pre-generated content, etc.


They can if they know what they are doing. E.g. Reddit can push 
indexes to their CDN (Cloudflare?) for commonly requested topics.

There are many many layers and options for caching or delaying 
recomputation (e.g. eventual consistency strategies).


 using python affect the performance of your website.


No. You can do a lot of computation in Python in 10ms. Using 
Python over C++/Go affects the number of instances, not the 
performance.

You don't handle one request at the time, so if you use Python 
you handle fewer concurrent requests than in Go. That is the only 
difference unless you are doing something real time (like a game 
server).
Oct 14 2015
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Wednesday, 14 October 2015 at 18:55:28 UTC, Ola Fosheim 
Grøstad wrote:

 On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:

 websites? nope. like booking.com, airbnb.com, reddit.com are 
 popular websites that have many parts which have to be dynamic 
 and responsive as hell and they cannot use caching, 
 pre-generated content, etc.


 They can if they know what they are doing. E.g. Reddit can push 
 indexes to their CDN (Cloudflare?) for commonly requested 
 topics.



And that is exactly what reddit does:

«
cache-control:max-age=0, must-revalidate
cf-cache-status:HIT
cf-ray:23558623ce9b231e-FRA
»

As you can see my request for /r/programming at reddit.com found 
a hit in their CDN cache Cloudflare...

CDNs may allow explicit preloading and removal of outdated 
resources from caches (contrary to HTTP).
Oct 14 2015
prev sibling parent reply Chris <wendlec tcd.ie> writes:
On Wednesday, 14 October 2015 at 18:37:40 UTC, Mengu wrote:

 On Wednesday, 14 October 2015 at 05:42:12 UTC, Ola Fosheim 
 Grøstad wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 That's flaimbait:

 «Many really popular websites use Python. But why is that? 
 Doesn't it affect the performance of the website?»

 No. Really popular websites use pre-generated content / front 
 end caches / CDNs or wait for network traffic from distributed 
 databases.


 really popular portals, news sites? yes. really popular 
 websites? nope. like booking.com, airbnb.com, reddit.com are 
 popular websites that have many parts which have to be dynamic 
 and responsive as hell and they cannot use caching, 
 pre-generated content, etc.

 using python affect the performance of your website. if you 
 were to use ruby or php your web app would be slower than it's 
 python version. and python version would be slower than go or d 
 version.


Yep. This occurred to me too. Sorry Ola, but I think you don't 
know how sausages are made. Do you really think that all the 
websites out there are performance tuned by network programming 
specialists? You'd be surprised!
Oct 15 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 15 October 2015 at 09:24:52 UTC, Chris wrote:

 Yep. This occurred to me too. Sorry Ola, but I think you don't 
 know how sausages are made.


I most certainly do. I am both doing backend programming and we 
have a farm... :-)


 Do you really think that all the websites out there are 
 performance tuned by network programming specialists? You'd be 
 surprised!


If they are to scale, then they have to pick algorithms and 
architectures that scale. This is commodity nowadays. You want to 
get as close to O(1) as possible for requests. This is how you 
build scalable systems. No point in having 1ms response time 
under low load and 10000ms response time when the incoming link 
is saturated.

You'd rather have 100ms response under low load and 120ms 
response time when saturated + 99.9999% availability/uptime.

Robustness and scaling costs latency, but you want acceptable and 
stable QoS, not brilliant QoS under low load and horrible QoS 
under high load.

Scalable websites aren't designed like sportcars, they are 
designed like trains.
Oct 15 2015
parent reply Chris <wendlec tcd.ie> writes:
On Thursday, 15 October 2015 at 09:47:56 UTC, Ola Fosheim Grøstad 
wrote:

 On Thursday, 15 October 2015 at 09:24:52 UTC, Chris wrote:

 Yep. This occurred to me too. Sorry Ola, but I think you don't 
 know how sausages are made.


 I most certainly do. I am both doing backend programming and we 
 have a farm... :-)


Well, you know how gourmet sausages are made (100% meat), because 
you make them yourself apparently. But I was talking about the 
sausages you get out there ;) A lot of websites are not 
"planned". They are quickly put together to promote an idea. The 
code/architecture is not important at that stage. The idea is 
important. The website has to have dynamic content that can be 
edited by non-programmers (Not even PHP! HTML at most!). If you 
designed a website from a programming point of view first, you'd 
never get the idea out in time.
Oct 15 2015
next sibling parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Thu, 2015-10-15 at 10:00 +0000, Chris via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 Well, you know how gourmet sausages are made (100% meat), because=20
 you make them yourself apparently. But I was talking about the=20
 sausages you get out there ;) A lot of websites are not=20
 "planned". They are quickly put together to promote an idea. The=20
 code/architecture is not important at that stage. The idea is=20
 important. The website has to have dynamic content that can be=20
 edited by non-programmers (Not even PHP! HTML at most!). If you=20
 designed a website from a programming point of view first, you'd=20



And most commercial websites selling things are truly appalling: slow
performance, atrocious usability/UX. Who cares if the site is
brilliantly tuned if it is unusable?


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Oct 15 2015
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 15 October 2015 at 10:00:21 UTC, Chris wrote:

 about the sausages you get out there ;) A lot of websites are 
 not "planned". They are quickly put together to promote an idea.


They are WordPress sites... :-(


 If you designed a website from a programming point of view 
 first, you'd never get the idea out in time.


It's not that bad, but modelling data for nosql databases is a 
bigger challenge than getting decent performance from the code.

There is another issue with using languages like Rust/C++/D and 
that is: if it crashes you loose all the concurrent requests, 
perhaps even without a reasonable log trace. What I'd want for 
handling requests is something less fragile where only the single 
request that went bad crash out. Pure Python and Java provide 
this property.
Oct 15 2015
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


I was just writing some R code yesterday after playing around 
with D for a couple weeks. I accomplished more in an afternoon of 
R coding than I think I had in like a month's worth of playing 
around with D. The same is true for python.
Oct 14 2015
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I was just writing some R code yesterday after playing around 
 with D for a couple weeks. I accomplished more in an afternoon 
 of R coding than I think I had in like a month's worth of 
 playing around with D. The same is true for python.


As someone who uses both D and Python every day, I find that - 
once you are proficient in both - initial productivity is higher 
in Python and then D starts to overtake as a project gets larger 
and/or has stricter requirements. I hope never to have to write 
anything longer than a thousand lines in Python ever again.
Oct 14 2015
next sibling parent reply David DeWitt <dkdewitt gmail.com> writes:
On Wednesday, 14 October 2015 at 14:48:22 UTC, John Colvin wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I was just writing some R code yesterday after playing around 
 with D for a couple weeks. I accomplished more in an afternoon 
 of R coding than I think I had in like a month's worth of 
 playing around with D. The same is true for python.


 As someone who uses both D and Python every day, I find that - 
 once you are proficient in both - initial productivity is 
 higher in Python and then D starts to overtake as a project 
 gets larger and/or has stricter requirements. I hope never to 
 have to write anything longer than a thousand lines in Python 
 ever again.


That's true until you need to connect to other systems.  There 
are countless clients built for other systems thats are used in 
real world applications.  With web development the Python code 
really just becomes glue nowadays and api's.  I understand D is 
faster until you have to build the clients for systems to 
connect.  We have an application that uses Postgres, 
ElasticSearch, Kafka, Redis, etc. This is plenty fast and the 
productivity of Python is more than D as the clients for 
Elasticsearch, Postgres and various other systems are unavailable 
or incomplete.  Sure D is faster but when you have other real 
world systems to connect to and time constraints on projects how 
can D be more productive or faster?  Our python code essentially 
becomes the API and usage of clients to other systems which 
handle a majority of the hardcore processing.  Once D gets 
established with those clients and they are battle tested then I 
will agree.  To me productivity is more than the language itself 
but also building real world applications in a reasonable 
time-frame.  D will get there but is nowhere near where Python is.
Oct 14 2015
next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 14 October 2015 at 15:25:22 UTC, David DeWitt wrote:

 On Wednesday, 14 October 2015 at 14:48:22 UTC, John Colvin 
 wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I was just writing some R code yesterday after playing around 
 with D for a couple weeks. I accomplished more in an 
 afternoon of R coding than I think I had in like a month's 
 worth of playing around with D. The same is true for python.


 As someone who uses both D and Python every day, I find that - 
 once you are proficient in both - initial productivity is 
 higher in Python and then D starts to overtake as a project 
 gets larger and/or has stricter requirements. I hope never to 
 have to write anything longer than a thousand lines in Python 
 ever again.


 That's true until you need to connect to other systems.  There 
 are countless clients built for other systems thats are used in 
 real world applications.  With web development the Python code 
 really just becomes glue nowadays and api's.  I understand D is 
 faster until you have to build the clients for systems to 
 connect.  We have an application that uses Postgres, 
 ElasticSearch, Kafka, Redis, etc. This is plenty fast and the 
 productivity of Python is more than D as the clients for 
 Elasticsearch, Postgres and various other systems are 
 unavailable or incomplete.  Sure D is faster but when you have 
 other real world systems to connect to and time constraints on 
 projects how can D be more productive or faster?  Our python 
 code essentially becomes the API and usage of clients to other 
 systems which handle a majority of the hardcore processing.  
 Once D gets established with those clients and they are battle 
 tested then I will agree.  To me productivity is more than the 
 language itself but also building real world applications in a 
 reasonable time-frame.  D will get there but is nowhere near 
 where Python is.


Python is inherently quite good for glue and has great library 
support, so if that's the majority of your work then Python is a 
good choice. On the other hand, there's plenty of programming out 
there that isn't like that.
Oct 14 2015
parent reply David DeWitt <dkdewitt gmail.com> writes:
On Wednesday, 14 October 2015 at 15:31:49 UTC, John Colvin wrote:

 On Wednesday, 14 October 2015 at 15:25:22 UTC, David DeWitt 
 wrote:

 On Wednesday, 14 October 2015 at 14:48:22 UTC, John Colvin 
 wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I was just writing some R code yesterday after playing 
 around with D for a couple weeks. I accomplished more in an 
 afternoon of R coding than I think I had in like a month's 
 worth of playing around with D. The same is true for python.


 As someone who uses both D and Python every day, I find that 
 - once you are proficient in both - initial productivity is 
 higher in Python and then D starts to overtake as a project 
 gets larger and/or has stricter requirements. I hope never to 
 have to write anything longer than a thousand lines in Python 
 ever again.


 That's true until you need to connect to other systems.  There 
 are countless clients built for other systems thats are used 
 in real world applications.  With web development the Python 
 code really just becomes glue nowadays and api's.  I 
 understand D is faster until you have to build the clients for 
 systems to connect.  We have an application that uses 
 Postgres, ElasticSearch, Kafka, Redis, etc. This is plenty 
 fast and the productivity of Python is more than D as the 
 clients for Elasticsearch, Postgres and various other systems 
 are unavailable or incomplete.  Sure D is faster but when you 
 have other real world systems to connect to and time 
 constraints on projects how can D be more productive or 
 faster?  Our python code essentially becomes the API and usage 
 of clients to other systems which handle a majority of the 
 hardcore processing.  Once D gets established with those 
 clients and they are battle tested then I will agree.  To me 
 productivity is more than the language itself but also 
 building real world applications in a reasonable time-frame.  
 D will get there but is nowhere near where Python is.


 Python is inherently quite good for glue and has great library 
 support, so if that's the majority of your work then Python is 
 a good choice. On the other hand, there's plenty of programming 
 out there that isn't like that.


I agree but the quora question ask why it is popular despite 
being slow and this is the reason.  If you are doing tasks that 
are computationally expensive in Python then yes it will be slow 
but Python is popular largely because of their web frameworks and 
support.  Even something like Pandas is good enough for most 
peoples data sets.  But still I think most people use it as glue 
and if they need something done they can pass it off to something 
else to do the "real" work.  If this wasn't the case then Python 
would not be as popular.  You pick the right tool for the right 
job maybe D and maybe Python and this doesn't mean your results 
will be slow.
Oct 14 2015
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Wednesday, 14 October 2015 at 15:49:20 UTC, David DeWitt wrote:

 I agree but the quora question ask why it is popular despite 
 being slow and this is the reason.  If you are doing tasks that 
 are computationally expensive in Python then yes it will be 
 slow but Python is popular largely because of their web 
 frameworks and support.


If it is slow then one can configure the load balancer to give 
more resources to that URL, or even run it on an insanely fast 
compute node with lots of memory for caching/memoing.

Or just implement that single high frequency URL handler in a 
different language if it scaling up resources become costly.


 You pick the right tool for the right job maybe D and maybe 
 Python and this doesn't mean your results will be slow.


Yes, you pick the right tool for the job, and in a web context 
that means picking a tool that has infrastructure support and 
makes it possible to cover future unexpected needs in a cost 
efficient and timely manner where you offload as much maintenance 
costs as possible to the infrastructure maintainers rather than 
the application maintainers.

I use AppEngine and Python is by far the most pleasant choice 
when choosing between Java, Go, Php and Python. For many very 
good reasons. But you can use whatever language you want for any 
particular URL so using Python is not a lock-in solution.

Using Rust or D for web developement do imply lockin on many 
dimensions (infrastructure, libraries, people, maintenance) 
compared to Python, Java and Ruby.
Oct 14 2015
prev sibling parent Laeeth Isharc <spamnolaeeth nospamlaeeth.com> writes:
On Wednesday, 14 October 2015 at 15:25:22 UTC, David DeWitt wrote:

 On Wednesday, 14 October 2015 at 14:48:22 UTC, John Colvin 
 wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I was just writing some R code yesterday after playing around 
 with D for a couple weeks. I accomplished more in an 
 afternoon of R coding than I think I had in like a month's 
 worth of playing around with D. The same is true for python.


 As someone who uses both D and Python every day, I find that - 
 once you are proficient in both - initial productivity is 
 higher in Python and then D starts to overtake as a project 
 gets larger and/or has stricter requirements. I hope never to 
 have to write anything longer than a thousand lines in Python 
 ever again.


 That's true until you need to connect to other systems.  There 
 are countless clients built for other systems thats are used in 
 real world applications.  With web development the Python code 
 really just becomes glue nowadays and api's.  I understand D is 
 faster until you have to build the clients for systems to 
 connect.  We have an application that uses Postgres, 
 ElasticSearch, Kafka, Redis, etc. This is plenty fast and the 
 productivity of Python is more than D as the clients for 
 Elasticsearch, Postgres and various other systems are 
 unavailable or incomplete.  Sure D is faster but when you have 
 other real world systems to connect to and time constraints on 
 projects how can D be more productive or faster?  Our python 
 code essentially becomes the API and usage of clients to other 
 systems which handle a majority of the hardcore processing.  
 Once D gets established with those clients and they are battle 
 tested then I will agree.  To me productivity is more than the 
 language itself but also building real world applications in a 
 reasonable time-frame.  D will get there but is nowhere near 
 where Python is.


Few thoughts:

1. It's easy to embed Python in your D applications.  I do this 
for things like web scraping and when I want to write something 
quick to read simple XML (I just convert to JSON).

2. Of course there is a Redis client.  Elasticsearch is an 
amazing product, but hardly requires much work to have a complete 
API.  I made a start on this, and if I use Elasticsearch more 
then I'll have one done and will release it.  I don't know the 
finer aspects of Postgres to know what is involved.

3. That raises a broader point, which is that it depends on the 
ultimate aim of your project and what it is about the right 
tradeoff between different things.  It will ultimately be much 
more productive for me to do things in D for the reasons John 
alludes to.  A little work to get started is neither here nor 
there in the major scheme of things.  Adam Ruppe made the same 
point - it's not all that much work to put a foundation that 
suits you in place.  You do it once (and maybe add things when 
something like Elasticsearch comes out), and that's it, apart 
from minor updates.  The dollar expenditure on building these 
things is not enormous given the stakes involved for me.  But 
that doesn't mean that you should get to the same answer, as it 
depends.

4. I am not sure that all web development is just glue, or will 
be going forward given what might be on the horizon, but time 
will tell.


Laeeth.
Oct 15 2015
prev sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Wed, 2015-10-14 at 14:48 +0000, John Colvin via Digitalmars-d-learn
wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc=20
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-s
 low
 Andrei suggested posting more widely.


=20
 I was just writing some R code yesterday after playing around=20
 with D for a couple weeks. I accomplished more in an afternoon=20
 of R coding than I think I had in like a month's worth of=20
 playing around with D. The same is true for python.


=20
 As someone who uses both D and Python every day, I find that -=20
 once you are proficient in both - initial productivity is higher=20
 in Python and then D starts to overtake as a project gets larger=20
 and/or has stricter requirements. I hope never to have to write=20
 anything longer than a thousand lines in Python ever again.


The thing about Python is NumPy, SciPy, Pandas, Matplotlib, IPython,
Jupyter, GNU Radio. The data science, bioinformatics, quant, signal
provessing, etc. people do not give a sh!t which language they used,
what they want is to get their results as fast as possible. Most of
them do not write programs that are to last, they are effectively throw
away programs. This leads them to Python (or R) and they are not really
interested in learning anything else.

The fact that NumPy sort of sucks in terms of performance, isn't
noticed by them
as they get their results "fast enough" and a lot faster than
sequential Python. The fact that if they used Chapel or even D for
their compute intensive code they would rapidly discover that NumPy
sort of sucks never really occurs to these people as they are focussed
on the results not the means of achieving them.

Polyglot Python/D or Python/Chapel with Matplotlib is the way to go.
But that really requires a D replacement for Pandas.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Oct 14 2015
next sibling parent Chris <wendlec tcd.ie> writes:
On Wednesday, 14 October 2015 at 18:17:29 UTC, Russel Winder 
wrote:


 The thing about Python is NumPy, SciPy, Pandas, Matplotlib, 
 IPython, Jupyter, GNU Radio. The data science, bioinformatics, 
 quant, signal provessing, etc. people do not give a sh!t which 
 language they used, what they want is to get their results as 
 fast as possible. Most of them do not write programs that are 
 to last, they are effectively throw away programs. This leads 
 them to Python (or R) and they are not really interested in 
 learning anything else.


Scary, but I agree with you again. In science this is exactly 
what usually happens. Throw away programs, a list here, a loop 
there, clumsy, inefficient code. And that's fine, in a way that's 
what scripting is for. The problems start to kick in when the 
same guys get the idea to go public and write a program that 
everyone can use. Then you have a mess of slow code 
(undocumented) in a slow language. This is why I always say "Use 
C, C++ or D from the very beginning" or at least document your 
code in a way that it can easily be rewritten in D or C. But 
well, you know, results, papers, conferences ... This is why many 
innovations live in an eternal Matlab or Python limbo.
Oct 15 2015
prev sibling parent Laeeth Isharc <laeeth nospamlaeeth.com> writes:
On Wednesday, 14 October 2015 at 18:17:29 UTC, Russel Winder 
wrote:

 On Wed, 2015-10-14 at 14:48 +0000, John Colvin via 
 Digitalmars-d-learn wrote:

 On Wednesday, 14 October 2015 at 14:32:00 UTC, jmh530 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-s
 low
 Andrei suggested posting more widely.


 
 I was just writing some R code yesterday after playing 
 around with D for a couple weeks. I accomplished more in an 
 afternoon of R coding than I think I had in like a month's 
 worth of playing around with D. The same is true for python.


 
 As someone who uses both D and Python every day, I find that - 
 once you are proficient in both - initial productivity is 
 higher in Python and then D starts to overtake as a project 
 gets larger and/or has stricter requirements. I hope never to 
 have to write anything longer than a thousand lines in Python 
 ever again.


 The thing about Python is NumPy, SciPy, Pandas, Matplotlib, 
 IPython, Jupyter, GNU Radio. The data science, bioinformatics, 
 quant, signal provessing, etc. people do not give a sh!t which 
 language they used, what they want is to get their results as 
 fast as possible. Most of them do not write programs that are 
 to last, they are effectively throw away programs. This leads 
 them to Python (or R) and they are not really interested in 
 learning anything else.

 The fact that NumPy sort of sucks in terms of performance, isn't
 noticed by them
 as they get their results "fast enough" and a lot faster than
 sequential Python. The fact that if they used Chapel or even D 
 for
 their compute intensive code they would rapidly discover that 
 NumPy
 sort of sucks never really occurs to these people as they are 
 focussed
 on the results not the means of achieving them.

 Polyglot Python/D or Python/Chapel with Matplotlib is the way 
 to go. But that really requires a D replacement for Pandas.



Russell, thanks for your thoughts - I appreciate it.

What would a Pandas replacement look like in D?
Oct 17 2015
prev sibling next sibling parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


I am coming at D by way of R, C++, Python etc. so I speak as a 
statistician who is interested in data science applications.

It's about programmer time. You have to weight the time it takes 
you to do the task in each programming language, if you are doing 
statistical analysis now, R and Python come out streaks ahead.

The scope roughly speaking is Research -> Deployment. R and 
Python sit on the research side, and Python/JVM technologies sit 
on the deployment side (broadly speaking). The question is where 
does D sit? What should D's data science strategy be?

To sit on the deployment side, D needs to grow it's big 
data/noSQL infrastructure for a start, then hook into a whole 
ecosystem of analytic tools in an easy and straightforward 
manner. This will take a lot of work!

I believe it is easier and more effective to start on the 
research side. D will need:

1. A data table structure like R's data.frame or data.table. This 
is a dynamic data structure that represents a table that can have 
lots of operations applied to it. It is the data structure that 
separates R from most programming languages. It is what pandas 
tries to emulate. This includes text file and database i/o from 
mySQL and ODBC for a start.

2. Formula class : the ability to talk about statistical models 
using formulas e.g. y ~ x1 + x2 + x3 etc and then use these 
formulas to generate model matrices for input into statistical 
algorithms.

3. Solid interface to a big data database, that allows a D data 
table <-> database easily

4. Functional programming: especially around data table and array 
structures. R's apply(), lapply(), tapply(), plyr and now 
data.table(,, by = list()) provides powerful tools for data 
manipulation.

5. A factor data type:for categorical variables. This is easy to 
implement! This ties into the creation of model matrices.

6. Nullable types makes talking about missing data more 
straightforward and gives you the opportunity to code them into a 
set value in your analysis. D is streaks ahead of Python here, 
but this is built into R at a basic level.

If D can get points 1, 2, 3 many people would be all over D 
because it is a fantastic programming language and is wicked fast.
Oct 14 2015
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer 
wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.



 I believe it is easier and more effective to start on the 
 research side. D will need: [snip]


Great list, but tons of work!
Oct 14 2015
parent reply data pulverizer <data.pulverizer gmail.com> writes:
On Thursday, 15 October 2015 at 02:20:42 UTC, jmh530 wrote:

 On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer 
 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.



 I believe it is easier and more effective to start on the 
 research side. D will need: [snip]


 Great list, but tons of work!


A journey of a thousand miles ...

I tried to start creating a data table type object by 
investigating variantArray: 
http://forum.dlang.org/thread/hhzavwrkbrkjzfohczyq forum.dlang.org but hit the
snag that D is a static programming language and may not allow the kind of
behaviour you need for creating the same kind of behaviour you need in data
table - like objects.

I envisage such an object as being composed of arrays of vectors 
where each vector represents a column in a table as in R - easier 
for model matrix creation. Some people believe that you should 
work with arrays of tuple rows - which may be more big data 
friendly. I am not overly wedded to either approach.

Anyway it seems I have hit an inherent limitation in the 
language. Correct me if I am wrong. The data frame needs to have 
dynamic behaviour bind rows and columns and return parts of 
itself as a data table etc and since D is a static language we 
cannot do this.
Oct 14 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Thu, 2015-10-15 at 06:48 +0000, data pulverizer via Digitalmars-d-
learn wrote:

=20


[=E2=80=A6]

 A journey of a thousand miles ...


Exactly.


 I tried to start creating a data table type object by=20
 investigating variantArray:=20
 http://forum.dlang.org/thread/hhzavwrkbrkjzfohczyq forum.dlang.org
  but hit the snag that D is a static programming language and may not
 allow the kind of behaviour you need for creating the same kind of
 behaviour you need in data table - like objects.
=20
 I envisage such an object as being composed of arrays of vectors=20
 where each vector represents a column in a table as in R - easier=20
 for model matrix creation. Some people believe that you should=20
 work with arrays of tuple rows - which may be more big data=20
 friendly. I am not overly wedded to either approach.
=20
 Anyway it seems I have hit an inherent limitation in the=20
 language. Correct me if I am wrong. The data frame needs to have=20
 dynamic behaviour bind rows and columns and return parts of=20
 itself as a data table etc and since D is a static language we=20
 cannot do this.


Just because D doesn't have this now doesn't mean it cannot. C doesn't
have such capability but R and Python do even though R and CPython are
just C codes.

Pandas data structures rely on the NumPy n-dimensional array
implementation, it is not beyond the bounds of possibility that that
data structure could be realized as a D module.

Is R's data.table written in R or in C? In either case, it is not
beyond the bounds of possibility that that data structure could be
realized as a D module.

The core issue is to have a seriously efficient n-dimensional array
that is amenable to data parallelism and is extensible. As far as I am
aware currently (I will investigate more) the NumPy array is a good
native code array, but has some issues with data parallelism and Pandas
has to do quite a lot of work to get the extensibility. I wonder how
the R data.table works.

I have this nagging feeling that like NumPy, data.table seems a lot
better than it could be. From small experiments D is (and also Chapel
is even more) hugely faster than Python/NumPy at things Python people
think NumPy is brilliant for. Expectations of Python programmers are
set by the scale of Python performance, so NumPy seems brilliant.
Compared to the scale set by D and Chapel, NumPy is very disappointing.
I bet the same is true of R (I have never really used R).

This is therefore an opportunity for D to step in. However it is a
journey of a thousand miles to get something production worthy.
Python/NumPy/Pandas have had a very large number of programmer hours
expended on them. =C2=A0Doing this poorly as a D modules is likely worse
than not doing it at all.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Oct 15 2015
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 15 October 2015 at 07:57:51 UTC, Russel Winder wrote:

 lot better than it could be. From small experiments D is (and 
 also Chapel is even more) hugely faster than Python/NumPy at 
 things Python people think NumPy is brilliant for. Expectations


Have you had a chance to look at PyOpenCL and PYCUDA?
Oct 15 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Thu, 2015-10-15 at 09:35 +0000, Ola Fosheim Gr=C3=B8stad via Digitalmars=
-
d-learn wrote:

 On Thursday, 15 October 2015 at 07:57:51 UTC, Russel Winder wrote:

 lot better than it could be. From small experiments D is (and=20
 also Chapel is even more) hugely faster than Python/NumPy at=20
 things Python people think NumPy is brilliant for. Expectations


=20
 Have you had a chance to look at PyOpenCL and PYCUDA?


Yes.=C2=A0

CUDA is of course doomed in the long run as Intel put GPGPU on the
processor chip. OpenCL will eventually be replaced with Vulkan
(assuming they can get the chips made).

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Oct 15 2015
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 15 October 2015 at 10:33:54 UTC, Russel Winder wrote:

 CUDA is of course doomed in the long run as Intel put GPGPU on 
 the processor chip. OpenCL will eventually be replaced with 
 Vulkan (assuming they can get the chips made).


I thought Vulkan was meant to replace OpenGL.
Oct 15 2015
parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Thu, 2015-10-15 at 17:00 +0000, jmh530 via Digitalmars-d-learn
wrote:

 On Thursday, 15 October 2015 at 10:33:54 UTC, Russel Winder wrote:

=20
 CUDA is of course doomed in the long run as Intel put GPGPU on=20
 the processor chip. OpenCL will eventually be replaced with=20
 Vulkan (assuming they can get the chips made).


=20
 I thought Vulkan was meant to replace OpenGL.


True, but there is an intent to try and have Vulkan allow for replacing
both OpenGL and OpenCL.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Oct 15 2015
prev sibling next sibling parent data pulverizer <data.pulverizer gmail.com> writes:
On Thursday, 15 October 2015 at 07:57:51 UTC, Russel Winder wrote:

 On Thu, 2015-10-15 at 06:48 +0000, data pulverizer via 
 Digitalmars-d- learn wrote:
 Just because D doesn't have this now doesn't mean it cannot. C 
 doesn't have such capability but R and Python do even though R 
 and CPython are just C codes.


I think the way R does this is that its dynamic runtime 
environment is used bind together native C basic type arrays. I 
wander if we could simulate dynamic behaviour by leveraging D's 
short compilation time to dynamically write/update data table 
source file(s) containing the structure of new/modified data 
tables?


 Pandas data structures rely on the NumPy n-dimensional array 
 implementation, it is not beyond the bounds of possibility that 
 that data structure could be realized as a D module.


Julia's DArray object is an interested take on this: 
https://github.com/JuliaParallel/DistributedArrays.jl

I believe that parallelism on arrays and data tables are 
different challenges. Data tables are easier since we can 
parallelise by row, thus the preference of having row-based 
tuples.


 The core issue is to have a seriously efficient n-dimensional 
 array that is amenable to data parallelism and is extensible. 
 As far as I am aware currently (I will investigate more) the 
 NumPy array is a good native code array, but has some issues 
 with data parallelism and Pandas has to do quite a lot of work 
 to get the extensibility. I wonder how the R data.table works.


R's data table is not currently parallelised


 I have this nagging feeling that like NumPy, data.table seems a 
 lot better than it could be. From small experiments D is (and 
 also Chapel is even more) hugely faster than Python/NumPy at 
 things Python people think NumPy is brilliant for. Expectations 
 of Python programmers are set by the scale of Python 
 performance, so NumPy seems brilliant. Compared to the scale 
 set by D and Chapel, NumPy is very disappointing. I bet the 
 same is true of R (I have never really used R).


Thanks for notifying me about Chapel - something else interesting 
to investigate. When it comes to speed R is very strange. Basic 
math (e.g. *, +, /) operation on an R array can be fast but 
for-looping will kill speed by hundreds of times - most things 
are slow in R unless they are directly baked into its base 
operations. You can write code in C and C++ can call it very 
easily in R though using its Rcpp interface.



 This is therefore an opportunity for D to step in. However it 
 is a journey of a thousand miles to get something production 
 worthy. Python/NumPy/Pandas have had a very large number of 
 programmer hours expended on them.  Doing this poorly as a D 
 modules is likely worse than not doing it at all.


I think D has a lot to offer the world of data science.
Oct 15 2015
prev sibling parent Laeeth Isharc <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 15 October 2015 at 07:57:51 UTC, Russel Winder wrote:

 On Thu, 2015-10-15 at 06:48 +0000, data pulverizer via 
 Digitalmars-d- learn wrote:

 


 […]

 A journey of a thousand miles ...


 Exactly.


 I tried to start creating a data table type object by
 investigating variantArray:
 http://forum.dlang.org/thread/hhzavwrkbrkjzfohczyq forum.dlang.org
  but hit the snag that D is a static programming language and 
 may not
 allow the kind of behaviour you need for creating the same 
 kind of
 behaviour you need in data table - like objects.
 
 I envisage such an object as being composed of arrays of 
 vectors where each vector represents a column in a table as in 
 R - easier for model matrix creation. Some people believe that 
 you should work with arrays of tuple rows - which may be more 
 big data friendly. I am not overly wedded to either approach.
 
 Anyway it seems I have hit an inherent limitation in the 
 language. Correct me if I am wrong. The data frame needs to 
 have dynamic behaviour bind rows and columns and return parts 
 of itself as a data table etc and since D is a static language 
 we cannot do this.


 Just because D doesn't have this now doesn't mean it cannot. C 
 doesn't have such capability but R and Python do even though R 
 and CPython are just C codes.

 Pandas data structures rely on the NumPy n-dimensional array 
 implementation, it is not beyond the bounds of possibility that 
 that data structure could be realized as a D module.

 Is R's data.table written in R or in C? In either case, it is 
 not beyond the bounds of possibility that that data structure 
 could be realized as a D module.

 The core issue is to have a seriously efficient n-dimensional 
 array that is amenable to data parallelism and is extensible. 
 As far as I am aware currently (I will investigate more) the 
 NumPy array is a good native code array, but has some issues 
 with data parallelism and Pandas has to do quite a lot of work 
 to get the extensibility. I wonder how the R data.table works.

 I have this nagging feeling that like NumPy, data.table seems a 
 lot better than it could be. From small experiments D is (and 
 also Chapel is even more) hugely faster than Python/NumPy at 
 things Python people think NumPy is brilliant for. Expectations 
 of Python programmers are set by the scale of Python 
 performance, so NumPy seems brilliant. Compared to the scale 
 set by D and Chapel, NumPy is very disappointing. I bet the 
 same is true of R (I have never really used R).

 This is therefore an opportunity for D to step in. However it 
 is a journey of a thousand miles to get something production 
 worthy. Python/NumPy/Pandas have had a very large number of 
 programmer hours expended on them.  Doing this poorly as a D 
 modules is likely worse than not doing it at all.


I think it's much better to start, which means solving your own 
problems in a way that is acceptable to you rather than letting 
perfection be the enemy of the good.  It's always easier to do 
something a second time too, as you learn from successes and 
mistakes and you have a better idea about what you want.  Of 
course it's better to put some thought into design early on, but 
that shouldn't end up in analysis paralysis.  John Colvin and 
others are putting quite a lot of thought into dlang science, it 
seems to me, but he is also getting stuff done.  Running D in a 
Jupyter notebook is something very useful.  It doesn't matter 
that it's cosmetically imperfect at this stage, and it won't stay 
that way.  And that's just a small step towards the bigger goal.
Oct 15 2015
prev sibling parent reply Laeeth Isharc <Laeeth.nospam nospam-laeeth.com> writes:
On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer 
wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I am coming at D by way of R, C++, Python etc. so I speak as a 
 statistician who is interested in data science applications.


Welcome...  Looks like we have similar interests.


 To sit on the deployment side, D needs to grow it's big 
 data/noSQL infrastructure for a start, then hook into a whole 
 ecosystem of analytic tools in an easy and straightforward 
 manner. This will take a lot of work!


Indeed.  The dlangscience project managed by John Colvin is very 
interesting.  It is not a pure stats project, but there will be 
many shared areas of need.  He has some v interesting ideas, and 
being able to mix Python and D in a Jupyter notebook is rather 
nice (you can do this already).

 I believe it is easier and more effective to start on the 
 research side. D will need:

 1. A data table structure like R's data.frame or data.table. 
 This is a dynamic data structure that represents a table that 
 can have lots of operations applied to it. It is the data 
 structure that separates R from most programming languages. It 
 is what pandas tries to emulate. This includes text file and 
 database i/o from mySQL and ODBC for a start.


I fully agree, and have made a very simple start on this.  See 
github. It's usable for my needs as they stand, although far from 
production ready or elegant.  You can read and write to/from CSV 
and HDF5.  I guess mysql and ODBC wouldn't be hard to add, but I 
don't myself need for now and won't have time to do myself.  If I 
have space I may channel some reesources in that direction some 
time next year.


 2. Formula class : the ability to talk about statistical models 
 using formulas e.g. y ~ x1 + x2 + x3 etc and then use these 
 formulas to generate model matrices for input into statistical 
 algorithms.


Sounds interesting.  Take a look at Colvin's dlang science draft 
white paper, and see what you would add.  It's a chance to shape 
things whilst they are still fluid.


 3. Solid interface to a big data database, that allows a D data 
 table <-> database easily


Which ones do you have in mind for stats?  The different choices 
seem to serve quite different needs.  And when you say big data, 
how big do you typically mean ?


 4. Functional programming: especially around data table and 
 array structures. R's apply(), lapply(), tapply(), plyr and now 
 data.table(,, by = list()) provides powerful tools for data 
 manipulation.


Any thoughts on what the design should look like?

To an extent there is a balance between wanting to explore data 
iteratively (when you don't know where you will end up), and 
wanting to build a robust process for production.  I have been 
wondering myself about using LuaJIT to strap together D building 
blocks for the exploration (and calling it based on a custom 
console built around Adam Ruppe's terminal).

 5. A factor data type:for categorical variables. This is easy 
 to implement! This ties into the creation of model matrices.

 6. Nullable types makes talking about missing data more 
 straightforward and gives you the opportunity to code them into 
 a set value in your analysis. D is streaks ahead of Python 
 here, but this is built into R at a basic level.


So matrices with nullable types within?  Is nan enough for you ?  
If not then could be quite expensive if back end is C.

 If D can get points 1, 2, 3 many people would be all over D 
 because it is a fantastic programming language and is wicked 
 fast.


What do you like best about it ?  And in your own domain, what 
have the biggest payoffs been in practice?
Oct 15 2015
parent data pulverizer <data.pulverizer gmail.com> writes:
On Thursday, 15 October 2015 at 21:16:18 UTC, Laeeth Isharc wrote:

 On Wednesday, 14 October 2015 at 22:11:56 UTC, data pulverizer 
 wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 I am coming at D by way of R, C++, Python etc. so I speak as a 
 statistician who is interested in data science applications.


 Welcome...  Looks like we have similar interests.


That's good to know


 To sit on the deployment side, D needs to grow it's big 
 data/noSQL infrastructure for a start, then hook into a whole 
 ecosystem of analytic tools in an easy and straightforward 
 manner. This will take a lot of work!


 Indeed.  The dlangscience project managed by John Colvin is 
 very interesting.  It is not a pure stats project, but there 
 will be many shared areas of need.  He has some v interesting 
 ideas, and being able to mix Python and D in a Jupyter notebook 
 is rather nice (you can do this already).


Thanks for bringing my attention to this, this looks interesting.


 Sounds interesting.  Take a look at Colvin's dlang science 
 draft white paper, and see what you would add.  It's a chance 
 to shape things whilst they are still fluid.


Good suggestion.


 3. Solid interface to a big data database, that allows a D 
 data table <-> database easily


 Which ones do you have in mind for stats?  The different 
 choices seem to serve quite different needs.  And when you say 
 big data, how big do you typically mean ?


What I mean is to start by tapping into current big data 
technologies. HDFS and Cassandra have C APIs which we can wrap 
for D.


 4. Functional programming: especially around data table and 
 array structures. R's apply(), lapply(), tapply(), plyr and 
 now data.table(,, by = list()) provides powerful tools for 
 data manipulation.


 Any thoughts on what the design should look like?


Yes, I think this is easy to implement but still important. The 



 To an extent there is a balance between wanting to explore data 
 iteratively (when you don't know where you will end up), and 
 wanting to build a robust process for production.  I have been 
 wondering myself about using LuaJIT to strap together D 
 building blocks for the exploration (and calling it based on a 
 custom console built around Adam Ruppe's terminal).


Sounds interesting


 6. Nullable types makes talking about missing data more 
 straightforward and gives you the opportunity to code them 
 into a set value in your analysis. D is streaks ahead of 
 Python here, but this is built into R at a basic level.


 So matrices with nullable types within?  Is nan enough for you 
 ?  If not then could be quite expensive if back end is C.


I am not suggesting that we pass nullable matrices to C 
algorithms, yes nan is how this is done in practice but you 
wouldn't have nans in your matrix at the point of modeling - 
they'll just propagate and trash your answer. Nullable types are 
useful in data acquisition and exploration - the more practical 
side of data handling. I was quite shocked to see them in D, when 
they are essentially absent from "high level" programming 
languages like Python. Real data is messy and having nullable 
types is useful in processing, storing and summarizing raw data. 

statistics working around them by using notional hacks. Nullables 

struggled with. The great news is that they are available in D so 
we can use them.


 If D can get points 1, 2, 3 many people would be all over D 
 because it is a fantastic programming language and is wicked 
 fast.


 What do you like best about it ?  And in your own domain, what 
 have the biggest payoffs been in practice?


I am playing with D at the moment. To become useful to me the 
data table structure is a must. I previously said points 1, 2, 
and 3 would get data scientists sucked into D. But the data table 
structure is the seed. A dynamic structure like that in D would 
catalyze the rest. Everything else is either wrappers, routine 
and maybe a lot of work but straightforward to implement. The 
data table structure for me is the real enigma.

The way that R's data types are structured around SEXPs is the 
key to all of this. I am currently reading through R's internal 
documentation to get my head around this.

https://cran.r-project.org/doc/manuals/r-release/R-ints.html
Oct 18 2015
prev sibling parent reply Namespace <rswhite4 gmail.com> writes:
On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


Maybe also interesting: 
https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1&slide=id.g70b0035b2_1_168
Oct 18 2015
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 18 October 2015 at 12:50:43 UTC, Namespace wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 Maybe also interesting: 
 https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1&slide=id.g70b0035b2_1_168


What I got out of that is that someone at Mozilla were writing a 
push service (stateful connections, which more demanding than 
regular http) and found that jitted Python was more suitable than 
Go for productivity reasons. Then they speculate that their own 
Rust will be better suited than Go for such services in the 
future, apparently not yet.


To the poster further up in the thread: turns out that reddit.com 
is implemented in Python and a little bit of C: 
https://github.com/reddit/reddit

So there we have it. Python gives higher productive at the cost 
of efficiency, but does not have a significant impact on 
effectiveness, for regular web services that are built to scale.
Oct 18 2015
next sibling parent reply Namespace <rswhite4 gmail.com> writes:
On Sunday, 18 October 2015 at 13:29:50 UTC, Ola Fosheim Grøstad 
wrote:

 On Sunday, 18 October 2015 at 12:50:43 UTC, Namespace wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 Maybe also interesting: 
 https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1&slide=id.g70b0035b2_1_168


 What I got out of that is that someone at Mozilla were writing 
 a push service (stateful connections, which more demanding than 
 regular http) and found that jitted Python was more suitable 
 than Go for productivity reasons. Then they speculate that 
 their own Rust will be better suited than Go for such services 
 in the future, apparently not yet.


I liked the fact that Python with PyPy is more performant than Go 
(in contrast to the title "Python is slow") and that Go-Routines 
leak.


 To the poster further up in the thread: turns out that 
 reddit.com is implemented in Python and a little bit of C: 
 https://github.com/reddit/reddit

 So there we have it. Python gives higher productive at the cost 
 of efficiency, but does not have a significant impact on 
 effectiveness, for regular web services that are built to scale.
Oct 18 2015
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 18 October 2015 at 13:57:40 UTC, Namespace wrote:

 I liked the fact that Python with PyPy is more performant than 
 Go (in contrast to the title "Python is slow") and that 
 Go-Routines leak.


Yes, Python apparently used less memory, which is rather 
important when you write a service with persistent websocket 
connections, like a webchat or game. Old school stackless 
coroutines probably would be better than fibers like D and Go 
uses.

An alternative to writing such code for the application is to get 
persistent connections by "ready made" server-infrastructure 
(which probably is more reliable anyway). On AppEngine you have 
something called channels which basically allows you to send 
messages to a connected client push-style:

https://cloud.google.com/appengine/docs/python/channel/

As far as I can tell that means that the application server can 
die without loosing the connection.
Oct 18 2015
prev sibling parent reply Mengu <mengukagan gmail.com> writes:
On Sunday, 18 October 2015 at 13:29:50 UTC, Ola Fosheim Grøstad 
wrote:

 On Sunday, 18 October 2015 at 12:50:43 UTC, Namespace wrote:

 On Tuesday, 13 October 2015 at 23:26:14 UTC, Laeeth Isharc 
 wrote:

 https://www.quora.com/Why-is-Python-so-popular-despite-being-so-slow
 Andrei suggested posting more widely.


 Maybe also interesting: 
 https://docs.google.com/presentation/d/1LO_WI3N-3p2Wp9PDWyv5B6EGFZ8XTOTNJ7Hd40WOUHo/mobilepresent?pli=1&slide=id.g70b0035b2_1_168


 What I got out of that is that someone at Mozilla were writing 
 a push service (stateful connections, which more demanding than 
 regular http) and found that jitted Python was more suitable 
 than Go for productivity reasons. Then they speculate that 
 their own Rust will be better suited than Go for such services 
 in the future, apparently not yet.


 To the poster further up in the thread: turns out that 
 reddit.com is implemented in Python and a little bit of C: 
 https://github.com/reddit/reddit

 So there we have it. Python gives higher productive at the cost 
 of efficiency, but does not have a significant impact on 
 effectiveness, for regular web services that are built to scale.


that's the pylons guy. he also has many python libraries for web 
development. reddit is built with pylons btw and pylons is now 
pyramid.

i've seen the presentation and i can't stop thinking how it'd be 
if they had chosen D instead of Go.
Oct 18 2015
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 18 October 2015 at 20:44:44 UTC, Mengu wrote:

 i've seen the presentation and i can't stop thinking how it'd 
 be if they had chosen D instead of Go.


Not much better, probably worse, given that Go has stack 
protection for fibers and D doesn't. So in Go you can get away 
with 2K growable stacks, in D you would need a lot more to stay 
on the safe side.

IIRC he claims that CPython would  fast enough for their 
application and that the application was memory limited and not 
computation limited.
Oct 18 2015